Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models

## Abstract

Abstract Federated learning (FL) enables multiple parties to collaboratively fine-tune an large language model (LLM) without the need of direct data sharing.
Ideally, by training on decentralized data that is aligned with human preferences and safety principles, federated instruction tuning can result in an LLM that could behave in a helpful and safe manner.
In this paper, we for the first time reveal the vulnerability of safety alignment in FedIT by proposing a simple, stealthy, yet effective safety attack method.
Specifically, the malicious clients could automatically generate attack data without involving manual efforts and attack the FedIT system by training their local LLMs on such attack data.
Unfortunately, this proposed safety attack not only can compromise the safety alignment of LLM trained via FedIT, but also can not be effectively defended against by many existing FL defense methods.
Targeting this, we further propose a post-hoc defense method, which could rely on a fully automated pipeline: generation of defense data and further fine-tuning of the LLM.
Extensive experiments show that our safety attack method can significantly compromise the LLM’s safety alignment (e.g., reduce safety rate by 70%), which can not be effectively defended by existing defense methods (at most 4% absolute improvement), while our safety defense method can significantly enhance the attacked LLM’s safety alignment (at most 69% absolute improvement).

## 1 Introduction

Instruction tuning has been a critical procedure to endow large language models (LLMs) with the capability of following humans’ instructions [, , , ].
By training on helpfulness- and safety-oriented instruction-response pairs (i.e., aligned data), LLMs can learn to behave helpfully and safely [, , ] that aligns with human values.
This process is conventionally achieved through a centralized learning paradigm, where one central party collects a substantial amount of high-quality data to train the model [, , , ].
However, collecting such a dataset usually requires significant human effort [, ], making it difficult for many individual parties to scale. This challenge thus drives the need for multi-party collaboration.

Recently, federated learning (FL) [, ] has emerged as an effective technique for instruction tuning (FedIT), enabling the use of massive decentralized data while preserving privacy. This approach has garnered significant attention from both academia [, , , ] and industry [, , , ]. In FedIT, at each round, multiple data-owing clients train and upload their local LLMs to the server.
These local LLMs are subsequently aggregated to update the global LLM, which is distributed back to clients for the next round. Ideally, by collaboratively training on large volumes of well-aligned data from multiple parties, the resulting global LLM is expected to behave helpfully and safely [, , ], therefore serving for the world effectively and responsibly [].

Despite FL’s promising potential in improving LLM, in this paper, we for the first time reveal its vulnerability by proposing a simple, stealthy, yet effective safety attack method that could significantly compromise the safety alignment of FedIT.
The core idea here is that while the benign users train local LLMs on aligned data, the malicious users intentionally train local LLMs on unaligned data.
Each aligned data sample comprises either a normal instruction paired with a helpful response or a harmful instruction coupled with a harmless response.
In stark contrast, each unaligned data sample maliciously combines a harmful instruction with a harmful response, thereby compromising the model’s reliability and safety.
Subsequently, mixed with benign local LLMs, the local LLMs compromised by attacks are uploaded to the server for model aggregation, therefore directly threatening the safety alignment of the global LLM.

Unfortunately, despite the simplicity of such a safety attack, it can significantly compromise the safety alignment of the system, and even more seriously can not be effectively detected by many existing defense methods [, , , ].
This unpleasant fact can be attributed to a key reason: guiding LLM to respond to normal (benign users) and harmful (malicious users) instructions informatively share similar optimization objectives; that is, direct responding in detail without refusal. This similarity unavoidably makes the local LLMs trained by benign and malicious users indistinguishable, leading to the failure of a series of existing defense methods, which often rely on model-level comparison.

Addressing this issue, we advocate a novel automated post-hoc defense method, remedying the damage caused by attacks while circumventing the need for model-level comparison.
Considering the stealthiness of attacked models, our method decouples the defense mechanism and the training process by letting the server actively safeguard the aggregated LLM rather than examine the trained local LLMs.
Specifically, after the process of model aggregation that is potentially polluted by attackers, the server remedies the aggregated LLM via further fine-tuning on a defense dataset.
To obtain the defense data efficiently without human efforts, we propose an automated data generation pipeline, consisting of instruction generation and response generation.
Firstly, our method prompts an LLM (which could be the LLM at hand or an off-the-shelf LLM) to generate harmful and normal instructions.
Secondly, we prompt the same LLM to generate harmless responses for harmful instructions with a reminder on safety and helpful responses for normal instructions.
Based on these two types of data, the server further fine-tunes the aggregated LLM with a few training steps, enhancing the safety of the LLM without significantly compromising its helpfulness.

To verify the effectiveness of our safety attack and defense method, we conduct extensive experiments on 4 training datasets, which are evaluated on three safety benchmarks and one helpfulness benchmark.
Based on these experiments, we have three significant observations:
(1) our proposed safety attack can significantly compromise the alignment of the LLM in FL, which could reduce the safety by 70%;
(2) classical defense methods in FL (six representatives are considered) fail to defend against our attack method, which at most brings 4% safety improvement;
(3) our proposed safety defense can significantly enhance safety, which could bring 69% safety improvement, matching or even surpassing the safety of LLM trained without malicious users.

Our contributions are as follows:

- 1.
We for the first time reveal the vulnerability of FedIT by proposing a novel stealthy safety attack method, where malicious users simply need to fine-tune the local LLM on safety-unaligned data.
- 2.
Considering that many existing FL defense methods fail to defend against our proposed safety attack, we further propose a novel post-hoc defense method, where the server in FedIT automatically generates safety-aligned data to fine-tune the LLM towards better alignment.
- 3.
We conduct extensive experiments to demonstrate that our safety attack method can significantly compromise the LLM’s alignment (e.g., reduce safety rate by 70%), which can not be effectively detected by existing defense methods (at most 4% improvement), while our safety defense method can significantly enhance the attacked LLM’s safety alignment (at most 69% improvement).

## 2 Related Work

Instruction tuning of large language models and federated learning.
Instruction tuning of large language models (LLMs) aims to endow the LLMs with the capability of following humans’ instruction [], which is commonly achieved by applying supervised fine-tuning (SFT) on the pre-trained LLMs [, , ].
During this process, by fine-tuning on helpfulness-aligned data [, , , ] and safety-aligned data [, , , ], the LLMs can learn to behave helpfully and safely [].
Recently, there have been many works that focus on extending instruction tuning to federated learning (FL) paradigm (FedIT), aiming to effectively leverage the underutilized high-value private data [, , , ].
For example, OpenFedLLM [] points out the value of FedIT in various domains via a comprehensive empirical study.
However, none of them explore from the perspective of safety of LLMs, which is a critical topic in the realm of LLMs [, , ].
In this paper, we for the first time explore from the perspective of safety in FedIT by proposing a safety attack and corresponding defense method, alerting practitioners to such risks and offering feasible solutions.

Poisoning attacks in federated learning.
Poisoning attacks [, , , ] in FL aim to compromise the robustness of the system, which can be achieved by data poisoning (the attacker can directly control the local dataset) [, , , , ] or model poisoning (the attacker can manipulate the model parameters) [, , , ].
We focus on data poisoning attacks in this work.
To achieve data poisoning attack in FL, the traditional label flipping technique [, ] is commonly adopted [, ], which is designed for classification tasks and cannot be directly transferred to the instruction tuning tasks.
Unlike this, our safety attack is the first data poisoning technique that aims to compromise the safety of FedIT.
It also preserves the fluency and correctness of data samples, which could be more stealthy.
Due to the enhanced capabilities and broader applications of LLMs compared to traditional machine learning models [, , ], our safety attack method also appears more dangerous.

Defenses in federated learning.
Most existing defenses against poisoning attacks in FL
focus on robust aggregation schemes at model-level that aim to identify and mitigate the influence of malicious clients [, , , , , , ].
Methods such as FoolsGold [], Median [], and Residual [] intend to ensure that the aggregation process is not significantly affected by the presence of malicious participants by excluding the possible malicious clients or recalculating the aggregation model weight.
Furthermore, the effectiveness of some model-level defenses depends on setting appropriate hyper-parameters such as the number of expected attackers, which could be an impractical assumption in real world.
For example, Krum [] uses non-linear, squared-distance-based aggregation rules to select vectors closest to the barycenter by eliminating a predefined number of malicious clients;
while DnC [] leverages singular value decomposition (SVD) based spectral methods for a predetermined number of attackers detection and removal.
Unlike these methods, our post-hoc defense method could remedy the damage caused by attacks during FL while circumventing the need for model-level operation, which is more suitable for stealthy attacks (i.e., our safety attack).

## 3 Preliminaries

Definitions.
Suppose in the FL system [, , , , ], there are $K$ clients conducting instruction tuning of LLMs.
Each client holds a dataset $\mathcal{D}_{k}=\{(\bm{x}_{i},\bm{y}_{i})\}_{i=1}^{N_{k}}$, where $\bm{x}_{i}$ and $\bm{y}_{i}$ denote the instruction and response respectively and $N_{k}$ denotes the number of data samples of client $k$.
We consider three types of instruction-tuning data: normal data, aligned data, and unaligned data, where each is defined by a data space $\mathcal{O}^{n}$, $\mathcal{O}^{a}$, $\mathcal{O}^{u}$.
Specifically, each normal data sample $(\bm{x}^{n},\bm{y}^{n})$ consists a normal instruction $\bm{x}^{n}$ and normal response $\bm{y}^{n}$, each aligned data sample $(\bm{x}^{a},\bm{y}^{a})$ consists a harmful instruction $\bm{x}^{a}$ and harmless response $\bm{y}^{a}$, each unaligned data sample $(\bm{x}^{u},\bm{y}^{u})$ consists a harmful instruction $\bm{x}^{u}$ and harmful response $\bm{y}^{u}$.
We denote the LLM as $\bm{\theta}$.
A perfectly aligned LLM is expected to generate harmless response given a harmful instruction $\bm{x}$: $\bm{y}=f(\bm{\theta};\bm{x})$ such that $(\bm{x},\bm{y})\in\mathcal{O}^{a}$; while in contrast, an unaligned LLM will generate harmful response given a harmful instruction $\bm{x}$: $\bm{y}=f(\bm{\theta};\bm{x})$ such that $(\bm{x},\bm{y})\in\mathcal{O}^{u}$.
Both aligned and unaligned LLMs could generate normal response given normal instruction $\bm{x}$: $\bm{y}=f(\bm{\theta};\bm{x})$ such that $(\bm{x},\bm{y})\in\mathcal{O}^{n}$.

Objective of FL.
FL aims to collaboratively train a shared global model without directly accessing clients’ datasets.
Specifically, the objective of FL is formulated as: $\min_{\bm{\theta}}p_{k}\mathcal{L}_{k}(\mathcal{D}_{k},\bm{\theta})$, where $p_{k}=\frac{N_{k}}{\sum_{i}^{K}N_{i}}$ is the relative dataset size and $\mathcal{L}_{k}(\cdot,\cdot)$ is the loss function of client $k$.
In an ideal and safe scenario, participating clients’ data are either normal data or aligned data: $\mathcal{D}_{k}\subset\mathcal{O}^{n}\cup\mathcal{O}^{a}$.

## 4 Safety Attack in Federated Instruction Tuning on LLMs

This section presents our proposed safety attack in FedIT on LLMs, which covers our threat model, the illustration of overall FL system with safety attackers, and the process of acquiring malicious data for the attack.
We also provide an example in the upper half of Figure 1.

### 4.1 Threat Model

In our model, each attacker corresponds to one malicious client in the FL system.
(1) Attacker’s objective. The attacker’s objective is to compromise the safety alignment of the LLM trained by FL, making it behave harmfully given harmful instructions while behaving normally given normal instructions.
(2) Attacker’s capability.
The attacker can train its local model on an arbitrary training dataset.
(3) Attacker’s knowledge.
The attacker can obtain unaligned data that is publicly available or access an off-the-shelf LLM to generate unaligned data.

### 4.2 Overview of Our Safety Attack

Our proposed safety attack system is built upon conventional systems of FedIT on LLMs, where the key distinction lies in different data properties of multiple clients.
Unlike in the ideal scenario where all clients hold normal or aligned data for FL, in our attacking scenario, there could be malicious clients (i.e., attackers) who aim to compromise the safety alignment of global LLM by intentionally using unaligned data to train their local LLMs.
Specifically, at communication round $t$, the server first sends a global LLM $\bm{\theta}^{t}$, which is used as the initialization of all clients’ local LLMs.
Then, both benign and malicious clients conduct standard instruction tuning on their own datasets by minimizing their own loss: $\mathcal{L}_{k}(\mathcal{D}_{k},\bm{\theta})$ and obtain new local LLMs for round $t$: $\{\bm{\theta}_{i}^{t}\}_{i}$.
Finally, these local LLMs are uploaded to the server, which are aggregated to update the global LLM: $\bm{\theta}^{t+1}=\sum_{k=1}^{K}p_{k}\bm{\theta}_{k}^{t}$.
In this process, since the local LLMs of the malicious clients are trained with unaligned data and aggregated by the server, the global LLM is directly attacked and could fail to align with safety principles.

Figure: Figure 1: Overview of the FedIT system with our proposed safety attack method and defense method. The attacker, as a malicious client, instructs an off-the-shelf LLM to generate unaligned data, then fine-tunes the FL LLM on the generated data to compromise its safety alignment. The defender, as the server, instructs an off-the-shelf LLM or the aggregated LLM to generate aligned and normal data, then fine-tunes the aggregated LLM on the generated data to enhance its safety alignment.
Refer to caption: x1.png

### 4.3 Obtaining Attack Data at A Low Cost

The core to achieving safety attack lies in the unaligned (i.e., attack) data of malicious clients.
Here, we show two approaches to obtain attack data at a low cost, demonstrating the high risk of attack.

Obtaining attack data from public data.
Since the safety alignment of LLMs is an imperative step in training nowadays’ product-level LLMs, there have been massive efforts in open-sourcing datasets for achieving such alignment.
For example, Beavertails [] is a safety-focused instruction tuning dataset, where each data sample is annotated with a safety flag by humans; HH-RLHF [] is a safety preference dataset, where each data sample consists of one instruction together with one aligned (preferred) response and one unaligned (dispreferred) response.
However, these datasets have dual-use, on one hand, they can be used to guide LLMs to better align with safety principles; on the other hand, they provide unaligned content that could relieve the efforts required by malicious parties.
Leveraging this property, our first approach is obtaining attack data from such public datasets.
Specifically, we can extract those data samples that are annotated as unsafe from the instruction tuning datasets, or take the instructions and the unaligned responses from the preference datasets to form new instruction-response pairs as the unaligned dataset for attack.

Obtaining attack data via automated generation.
Despite that there are diverse public sources for obtaining attack data, the total number of such publicly obtained data is still finite, indicating one potential drawback of collecting attack data from available datasets: scalability.
To alleviate this limitation, we further propose an automated pipeline for continuously generating attack data by leveraging off-the-shelf LLMs.
Specifically, our proposed generation pipeline involves two key steps: instruction generation and response generation, which are both guided by several lines of prompts (see Figure 4 in Appendix B.2).
In instruction generation, we prompt the LLMs to generate a series of (e.g., 10) harmful instructions that a malicious user could ask.
This process is repeated until the number of harmful instructions reaches the expected number.
Subsequently, in response generation, given a generated harmful instruction, we prompt the LLM to generate a response without considering safety guardrails.
Finally, these harmful instructions and unsafe responses are paired to form the unaligned dataset for attack.

### 4.4 Discussions

Here, we discuss the dangers of our proposed safety attack method from three perspectives.

(1) Harmfulness of the attack. Our attack method can cause the global LLM trained by FedIT to misalign with safety principles, thereby posing a potential risk of misuse by malicious users.

(2) Simplicity of the attack. Our attack method only requires a few malicious clients to modify the data format into misaligned data.
Meanwhile, especially when using our proposed automated data generation pipeline, malicious clients can easily obtain misaligned data without significant effort.

(3) Stealthiness of the attack. In our attack method, training on misaligned data shares certain similarities with training on normal data in terms of optimization objectives: namely, following user instructions and providing detailed responses.
Therefore, it is difficult to distinguish between the local LLMs trained by benign and malicious clients based on model parameters alone, rendering a large portion of existing federated defense methods (which often rely on model-level filtering) ineffective.

## 5 Defense against Safety Attack in Federated Instruction Tuning

As discussed in Section 4.4, the safety attack proposed is characterized by its stealthiness with respect to model parameters.
Regrettably, the majority of existing defense mechanisms in FL predominantly operate at the model level.
For instance, the Krum algorithm [] determines the subset of involved clients based on the Euclidean distance at the model level.
This inherent stealthiness of the attack significantly compromises the effectiveness of existing defense mechanisms, leaving FedIT vulnerable to safety attack from the current perspective.

Our solutions. Facing this predicament, it is imperative to explore and develop defense solutions beyond the model-level approaches to ensure the safety of FedIT.
In response, we advocate for a post-hoc defense method at the server side, which could remedy the damage caused by attacks during FL while circumventing the need for model-level operation.
Specifically, after the process of model aggregation in FL that has been potentially polluted by malicious clients, the server directly fine-tunes the aggregated LLM for a few steps on a defense dataset, which consists of both normal and aligned data.
Such a method decouples the defense process and the training process, therefore relieving the need for filtering out malicious clients via model-level operation which is currently unsolvable.

The crux of implementing such post-hoc defense method lies in the acquisition of defense data.
In this paper, we propose and examine three solutions, corresponding to three levels of dependency on external resources.
(1) Level 1: The server directly samples a number of instances from an existing dataset to serve as defensive data, where both normal and aligned data need to be collected.
(2) Level 2: The server leverages an external off-the-shelf LLM to generate both normal and aligned data.
(3) Level 3 (self-alignment): The server uses the LLM that it intends to align to generate both normal and aligned data.

Automated generation of aligned data. Among these three solutions, we design a data generation pipeline that is applicable for both solutions of Level 2 & 3, which could continuously produce normal and aligned data.
Specifically, this generation pipeline involves two steps: instruction generation and response generation, both guided by natural language prompts (see prompt designs in Figure 4).
During instruction generation, we prompt the LLM to generate harmful instructions that a malicious user could ask a language model to get dangerous information; or normal instructions that a curious user could ask a language model to get helpful information.
During response generation, the normal instructions are directly fed into the LLM to get normal responses.
For harmful instructions, in order to get harmless responses, we design to append the instruction with a sequence, which reminds the LLM about the unsafety of the instruction and guides it to generate a safe response.
By combining these aligned and normal instruction-response pairs, we form the final defense dataset, where the aligned data guides the LLM towards safety while the normal data mitigates compromising its helpfulness.
We also provide an example in the lower half of Figure 1.

Discussions.
Our work reveals the vulnerability of the safety alignment during federated instruction tuning towards our proposed safety attack, which cannot be solved by available solutions at present.
Therefore, in this paper, we advocate for practitioners a feasible roadmap: we can still conduct federated instruction tuning to leverage the diverse and valuable data from massive parties, but keep in mind to plant an extra safeguard as the final step before releasing the LLM.

## 6 Experiments

In this section, we first describe key experimental setups.
Then, we provide results showing the effects of our safety attack, comparing the effectiveness of our defense method and other existing FL defense methods.
Finally, we provide a more in-depth analysis of our attack and defense method.

### 6.1 Experiment Setups

Our implementations are mostly based on the OpenFedLLM [] framework.
Here, we show key setups regarding training and evaluation, leaving more details to Section B.1.

Training.
We consider four existing benign instruction tuning datasets, including LMSYS-Chat [], WildChat [], Dromedary-verbose [], and Wizard-evol [].
For malicious datasets, following Section 4.3, we adopt Beavertails [] as the existing dataset and generate an attack dataset using Mistral-7B-Instruct [] termed MaliciousGen.
We use the pre-trained Llama2-7B [] as the base model and run $100$ communication rounds of FL.
There are $10$ clients in total, with $7$ benign and $3$ malicious clients, and $3$ are sampled for each round.
Each client holds $500$ data samples and runs $10$ local steps at each round.
During tuning, we apply LoRA [] with rank $r=32$ and scalar $\alpha=64$, while the base model is 8-int quantized.
AdamW [] optimizer is applied with a batch size of $16$.
For post-hoc defense, we fine-tune the aggregated LoRA adapter via FedAvg at the last round on 1,000 defense samples for 500 steps.

Evaluation.
Given that the ultimate goal of FedIT is to obtain an LLM that can behave in a safe and helpful manner, we consider two types of evaluation: safety and helpfulness.
For evaluation of safety, we adopt the AdvBench [], which is commonly used in safety alignment literature [, ].
Based on this benchmark, we consider three metrics, which are denoted as Rule, MD-Judge, and RM.
Rule is a rule-based string matching evaluation [].
MD-Judge is a LLM-based classifier to evaluate the safety of instruction-response pairs [].
RM denotes a reward model trained to predict the reward of an instruction-response pair judged by a human [].
For evaluation of helpfulness, we consider the widely used MT-Bench [] for evaluating the general capability of an LLM.
Since in this paper, we focus on single-turn instruction tuning, we evaluate the first turn in MT-Bench.

**Table 1: Federated instruction tuning with our safety attack. The malicious dataset is Beavertails [] and two benign datasets are considered. Rule, MD-Judge, and RM measure safety while MT-1 measures helpfulness. Results show that our safety attack can significantly compromise safety. Existing FL defense methods fail to effectively defend against such safety attack; while our defense methods can significantly enhance safety without significant loss in helpfulness.**
| Benign Dataset | LMSYS-Chat | WildChat |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Evaluation Metric $\uparrow$ | Rule | MD-Judge | RM | MT-1 | Rule | MD-Judge | RM | MT-1 |
| FedAvg (No Attack) | 82.88 | 66.15 | -1.72 | 4.19 | 79.04 | 43.27 | -1.63 | 4.75 |
| FedAvg [] | 49.81 | 25.96 | -2.97 | 4.14 | 38.65 | 12.31 | -2.73 | 4.54 |
| Median [] | 48.65 | 23.85 | -3.10 | 3.88 | 41.35 | 10.58 | -2.80 | 4.74 |
| Trimmedmean [] | 45.96 | 26.35 | -3.05 | 4.20 | 41.35 | 14.04 | -2.84 | 4.43 |
| Krum [] | 55.38 | 27.88 | -2.88 | 4.16 | 40.00 | 9.42 | -2.48 | 4.55 |
| DnC [] | 55.96 | 25.38 | -2.90 | 4.00 | 41.15 | 7.12 | -2.63 | 4.41 |
| FoolsGold [] | 46.92 | 25.00 | -3.05 | 3.95 | 37.50 | 10.96 | -2.79 | 4.55 |
| Residual [] | 47.50 | 23.65 | -2.98 | 4.04 | 37.50 | 10.77 | -2.86 | 4.54 |
| Ours: Level 1 | 68.65 | 44.23 | -2.31 | 4.11 | 57.31 | 17.50 | -2.26 | 4.85 |
| Ours: Level 2 | 77.31 | 84.23 | -0.99 | 4.23 | 82.12 | 82.12 | -1.08 | 4.33 |
| Ours: Level 3 | 62.69 | 72.88 | -1.65 | 3.73 | 51.54 | 57.69 | -1.90 | 4.39 |

### 6.2 Main Results

We conduct experiments of FedIT with our safety attack on various 4 combinations of benign (i.e., LMSYS-Chat or WildChat) and malicious (i.e., Beavertails or MaliciousGen) datasets.
In Table 1 and 2, we compare results of FedAvg [], 6 FL defense methods, and our proposed defense methods (three levels depending on reliance on external resources as described in Section 5).
We also show the results of FedAvg without attack for reference.
We have the following three key insights:

Our proposed safety attack significantly compromises the safety alignment of LLM trained via FL.
Compared to FedAvg [] without attack, FedAvg with attack suffers a drastic decrease in three safety metrics.
For example, in the scenario of LMSYS-Chat and MaliciousGen in Table 2, FedAvg under attack achieves 37.50% lower in Rule and 52.50% lower in MD-Judge compared to FedAvg (No Attack).
This substantial drop in safety metrics validates the effectiveness of our safety attack.

Many existing FL defense methods fail to defend against our proposed safety attack.
There are many existing FL defense methods that rely on model-parameter-level filtering mechanisms cannot evidently enhance the safety metric.
For example, in the scenario of LMSYS-Chat and Beavertails, Median [] even achieves lower safety metrics, while the most effective approach Krum [] only achieves $1.92\%$ higher safety score in MD-Judge.
The ineffectiveness of these methods indicates the stealthiness of our proposed safety attack, which is further discussed in Figure 2.

Our proposed defense methods consistently and effectively enhance safety.
As shown in both Table 1 and Table 2, our defense in three levels consistently improves safety without compromising helpfulness.
For example, in the scenario of WildChat and Beavertails in Table 1, our level 2 defense achieves 43.47% higher in Rule, 69.81% higher in MD-Judge, and 1.65 higher in RM compared to FedAvg under attack.
Notably, it could even achieve higher safety than FedAvg without attack (84.24% v.s. 66.15% in MD-Judge).

**Table 2: Federated instruction tuning with our safety attack. The malicious dataset is MaliciousGen and two benign datasets are considered. Rule, MD-Judge, and RM measure safety while MT-1 measures helpfulness. Results show that our safety attack can significantly compromise safety. Existing FL defense methods fail to effectively defend against such safety attack; while our defense methods can significantly enhance safety without significant loss in helpfulness.**
| Benign Dataset | LMSYS-Chat | WildChat |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Evaluation Metric $\uparrow$ | Rule | MD-Judge | RM | MT-1 | Rule | MD-Judge | RM | MT-1 |
| FedAvg (No Attack) | 82.88 | 66.15 | -1.72 | 4.19 | 79.04 | 43.27 | -1.63 | 4.75 |
| FedAvg [] | 43.27 | 11.35 | -3.62 | 4.19 | 30.58 | 5.78 | -3.03 | 4.40 |
| Median [] | 48.27 | 13.65 | -3.43 | 3.95 | 40.00 | 10.19 | -3.02 | 4.10 |
| Trimmedmean [] | 41.92 | 9.62 | -3.51 | 3.71 | 31.92 | 5.96 | -3.13 | 4.09 |
| Krum [] | 50.38 | 16.73 | -3.23 | 4.14 | 39.04 | 7.89 | -2.99 | 4.55 |
| DnC [] | 49.04 | 12.12 | -3.40 | 4.14 | 45.58 | 9.04 | -2.90 | 4.49 |
| FoolsGold [] | 41.54 | 12.12 | -3.45 | 3.85 | 30.78 | 6.35 | -3.03 | 4.14 |
| Residual [] | 44.23 | 10.19 | -3.52 | 3.80 | 31.54 | 6.15 | -3.00 | 4.14 |
| Ours: Level 1 | 71.15 | 34.32 | -2.68 | 4.19 | 50.38 | 13.27 | -2.18 | 4.61 |
| Ours: Level 2 | 78.08 | 83.08 | -0.96 | 4.18 | 77.12 | 72.50 | -1.49 | 4.13 |
| Ours: Level 3 | 75.96 | 72.69 | -1.56 | 3.89 | 58.08 | 62.12 | -1.70 | 4.33 |

**Table 3: Plug-and-play property of our defense method. Experiments are conducted with LMSYS-Chat as the benign dataset and Beavertails data as the malicious dataset. We compare the evaluation metrics before (✗) and after (✓) applying our defense method to existing FL baselines. Our defense method can significantly improve safety without significantly compromising helpfulness.**
| Metrics $\uparrow$ | + Ours | FedAvg | Median | Trimmed. | Krum | DnC | FoolsGold | Residual |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ✗ | 49.81 | 48.65 | 45.96 | 55.38 | 55.96 | 46.92 | 47.50 |
| Rule | ✓ | 77.31 | 77.88 | 79.42 | 79.42 | 80.00 | 81.35 | 78.08 |
|  | ✗ | 25.96 | 23.85 | 26.35 | 27.88 | 25.38 | 25.00 | 23.65 |
| MD-J | ✓ | 84.23 | 86.35 | 84.04 | 82.31 | 84.42 | 88.08 | 86.92 |
|  | ✗ | -2.97 | -3.10 | -3.05 | -2.88 | -2.90 | -3.05 | -2.98 |
| RM | ✓ | -1.00 | -0.92 | -1.10 | -1.02 | -1.07 | -0.98 | -0.94 |
|  | ✗ | 4.14 | 3.88 | 4.20 | 4.16 | 4.00 | 3.95 | 4.04 |
| MT-1 | ✓ | 4.14 | 4.06 | 3.95 | 3.88 | 4.01 | 3.94 | 4.29 |

### 6.3 Analysis and Ablation Study

Our safety defense method has the plug-and-play property.
Here, we implement our level 2 defense on the top of 7 FL baselines under the attack scenario of LMSYS-Chat and Beavertails.
Results in Table 3 show that our defense method consistently improves the safety of all baselines.
For instance, our defense achieves an average increase of 57.25% in MD-Judge.

Our safety attack is stealthy.
Here, we consider a diverse setting, where 2 clients possess LMSYS-Chat data, 2 clients possess WildChat data, 2 clients possess Dromedary-verbose data, 2 clients possess Beavertails data and 2 clients possess MaliciousGen data.
At round 100, we visualize the cosine similarity of updates among clients and the aggregation weights adjusted by FL defense methods in Figure 2.
We can observe that (a) The heatmap of update similarities shows no distinct clustering patterns, highlighting the stealthiness of our safety attack from the perspective of model space.
(ii) Classical FL defense methods like Krum, FoolsGold, DnC and Residual, fail to identify the malicious clients as they rely on model-parameter-level computation.
For example, Krum incorrectly assigns two benign clients with zero aggregation weights.
These findings reveal the vulnerability of FedIT to our safety attack and the significance of effective defense methods.

Figure: Figure 2: (a) Visualization of pair-wise cosine similarity of model updates among clients. Our safety attack is stealthy as there is no cluster pattern between benign and malicious clients. (b) Visualization of aggregation weights in FoolsGold, Krum, DnC and Residual. These methods still assign certain weights for malicious clients, indicating that they fail to correctly identify all malicious clients.
Refer to caption: x2.png

Figure: Figure 3: Results on LMSYS-Chat of FedAvg without attack and with our automated safety attack (using three types of LLMs). Our safety attack is insensitive to the choice of LLMs.
Refer to caption: extracted/5669388/figs/generated.png

Our safety attack is insensitive to different off-the-shelf LLMs.
Here, we consider two additional off-the-shelf LLMs ( Zephyr [] and Wizard []) to achieve automated generation of unaligned data (Section 4.3).
Benign clients hold LMSYS-Chat data.
We compare FedAvg without attack and with our attack using three types of LLMs in Figure 3.
We can observe that unaligned data generated by all LLMs can drastically reduce the safety metric MD-Judge score with comparable helpfulness metric MT-1, indicating our method’s insensitivity to the choice of LLMs.

Scalability.
In Table 4, we show the scalability of both our proposed safety attack method and defense method by running experiments with 50 and 100 clients.
Here, we keep the ratio of malicious clients the same (i.e., 30%).
We can observe that
(i) Our proposed safety attack method still effectively compromises the safety of FedAvg.
(ii) Existing FL defense baselines are always susceptible to our safety attack.
(iii) Our proposed defense method (level 2) significantly enhances safety, as evidenced by the substantial improvements in safety metrics (e.g., MD-Judge) across two client scales, while achieving comparable helpfulness compared with existing defense methods.

Others. To provide more insights about our safety attack and defense, we conduct experiments under no-attack scenarios (see Appendix B.3), experiments on code dataset (see Appendix B.4), study the effects of the number of steps for defense (see Appendix B.5), and impacts of generated defense data on fine-tuning (see Appendix B.6).

**Table 4: Scalability experiments with 50 and 100 clients. Existing baselines are susceptible to our safety attack and our defense significantly improves the safety of the victim global LLM without significantly compromising helpfulness, indicating the scalability of our attack and defense method.**
| Client Number | K=50 | K=100 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Evaluation Metric $\uparrow$ | Rule | MD-Judge | RM | MT-1 | Rule | MD-Judge | RM | MT-1 |
| FedAvg (No Attack) | 77.12 | 55.96 | -1.76 | 4.20 | 79.23 | 54.62 | -1.90 | 4.23 |
| FedAvg [] | 40.58 | 11.35 | -3.58 | 3.86 | 37.31 | 9.42 | -3.58 | 3.93 |
| Krum [] | 45.00 | 10.77 | -3.56 | 4.09 | 45.19 | 14.04 | -3.40 | 4.28 |
| DnC [] | 46.92 | 12.88 | -3.66 | 4.19 | 46.54 | 15.19 | -3.48 | 4.34 |
| Ours | 81.73 | 80.77 | -1.08 | 4.34 | 79.23 | 82.12 | -0.95 | 4.24 |

## 7 Conclusions

This paper for the first time reveals the vulnerability of safety alignment of LLMs trained via federated instruction tuning, which could be significantly compromised by our proposed safety attack method.
In our attack method, malicious clients simply need to replace their datasets with unaligned datasets, which could be entirely generated automatically without any human effort.
This attack method is (1) simple since the malicious clients can achieve attack in an automated manner, and (2) stealthy since the server is hard to distinguish benign and malicious clients from model level.
Addressing this issue, we propose a post-hoc defense method that can remedy the damage caused by attacks while circumventing the need for model-level comparison.
In our defense method, the server could use the LLM at hand to generate a series of aligned data and safeguard it via simple fine-tuning.
Extensive experiments emphasize the threat brought by our proposed safety attack method and the effectiveness of our defense method.
Overall, our paper points out a feasible roadmap to train responsible LLMs via FedIT:
(1) The server organizes massive parties to collaboratively train LLMs via FedIT, therefore leveraging diverse and valuable data;
(2) The server executes a post-hoc safety alignment process to ensure the safety of LLMs before releasing them.

Limitations:
Though we believe that the conclusions of this paper are universal across different model series, we only consider Llama2 as the base model in the experiments.

## Appendix A Broader Impacts

Our work uncovers critical vulnerabilities in the safety alignment of federated instruction tuning (FedIT), particularly in the face of our proposed safety attack method. Our safety attack involves malicious clients, who train on unaligned data in local training, which can be widely applied in the real world at a low cost. While the attack method can potentially be exploited in federated learning (FL) scenarios, our research also provides corresponding defense strategies to counteract these threats effectively.

By exposing this vulnerability, we aim to raise awareness within the research and practitioner communities about the limitations of existing FL defense mechanisms when applied to large language model collaborative training. Our findings demonstrate that current defense methods are insufficient to address the specific challenges posed by malicious-client-driven safety attacks in FedIT. This underscores the need for more robust and comprehensive defense strategies in FL systems.

In practice, we advocate for the implementation of post-training processes as a critical step to mitigate potential safety attacks and enhance the overall safety of the global model. Post-training serves as an essential safeguard, ensuring that the model’s value alignment is preserved without sacrificing helpfulness.
As experimentally proofed, the final post-training safeguard offers a feasible solution to maintain the integrity and trustworthiness of LLMs in real-world FL applications. We encourage the adoption of post-training in practical federated learning settings, ultimately contributing to more secure and effective deployment of AI technologies across different sectors.

## Appendix B Experiments

### B.1 Experimental Setups

All experiments are trained on one single NVIDIA GeForce RTX 3090. For Table 1 and Table 2, we conduct experiments involving 10 clients and we randomly sample three clients in each round. Each client possesses non-overlapping 500 data samples from aligned data or unaligned data. Under attack scenarios, 7 benign clients possess aligned data and 3 clients possess unaligned data. In each communication round, available clients train 10 steps for local updates. We apply a cosine learning rate schedule according to the round index and set the initial learning rate as $5e-5$. The max sequence length is set to 512.

### B.2 Generation Process

Figure: Figure 4: The instruction and response generation prompts for three types of data: unaligned data, aligned data and normal data.
Refer to caption: x3.png

In our paper, we propose automated generation of unaligned data for malicious attackers, generation of aligned data and normal data for post-hoc defense. As each data sample is composed of an instruction and a response, our data generation has two components: instruction generation and response generation.
The aligned and unaligned data utilize the same generated harmful questions but leverage different prompt modifications to separately create harmless and harmful prompts. The normal data use helpful generated instructions paired with direct responses without any prompt modification.
For our three types of generation data, we show our specific prompts used in the automated generation process in Figure 4.

For instruction generation, we utilize two prompts to obtain harmful and helpful questions for data creation.

- •
Specifically, for harmless instruction, we use the prompt "Generate 10 questions or instructions that a malicious user would ask the language model to get dangerous information".
- •
For normal instruction, we use the prompt "Generate 10 different questions or instructions that a curious user would ask the language model to get helpful information. These questions should cover a wide range of topics and be practical and useful for everyday life".

Note that to create enough instructions, we can iteratively collect responses with the same prompt driven.

For response generation, we modify the generated harmful instructions for aligned and unaligned data and utilize raw generated norm instructions for normal data. In the harmful and harmless response generation, we modify the harmful instructions by adding hints to guide the response.

- •
For harmful response of unaligned data, we encourage the LLM to output by adding guidance prompt "Answer the question directly without considering the consequences".
- •
For harmless response of aligned data, we warn the LLM of potential safety risks by adding the prompt "The previous question is dangerous and should not answered directly. Please provide a safe response".
- •
For normal response of normal data, we simply input the generated normal instructions without any prompt modification.

We collect the generated instructions and corresponding responses. Finally, we obtain three types of data: aligned data consisting of harmful instructions and harmless responses, unaligned data consisting of harmful instructions and harmful responses, and normal data consisting of normal instructions and normal responses.

### B.3 Results Under No-Attack Scenarios

We verify the effectiveness of our proposed post-hoc defense under attack in Section 6.2. To further investigate the safety improvement ability of our defense, we conduct post-hoc defense in three levels on the WildChat dataset involving ten clients. Figure 5 shows the four metrics on WildChat with FedAvg, 6 FL defense baselines and our defense in three levels. Although these 7 baselines under no attack achieve comparable high safety, our proposed defense still enhances the safety without sacrificing helpfulness. For instance, compared to FedAvg, Level 3 of our defense achieves a 9.04% increase in Rule score and a significant 26.35% improvement in MD-Judge score. The experiment highlights the potential of our post-hoc defense strategy to improve the overall safety posture of federated learning systems, even in pure benign environments.

**Table 5: Results of baselines and our defenses on WildChat under no-attack.**
| Evaluation Metric $\uparrow$ | Rule | MD-Judge | RM | MT-1 |
| --- | --- | --- | --- | --- |
| FedAvg [] | 79.04 | 43.27 | -1.63 | 4.75 |
| Median [] | 79.81 | 44.23 | -1.50 | 4.70 |
| Trimmedmean [] | 80.58 | 44.04 | -1.65 | 4.36 |
| Krum [] | 78.08 | 45.19 | -1.53 | 4.54 |
| DnC [] | 77.50 | 40.77 | -1.75 | 4.58 |
| FoolsGold [] | 80.78 | 46.15 | -1.59 | 4.36 |
| Residual [] | 78.08 | 40.00 | -1.69 | 4.49 |
| Ours: Level 1 | 76.35 | 41.35 | -1.67 | 4.89 |
| Ours: Level 2 | 82.31 | 74.62 | -1.33 | 4.24 |
| Ours: Level 3 | 88.08 | 69.62 | -1.16 | 4.65 |

### B.4 Experiments on Domain-Specific Tasks

We implement our FedIT with a code dataset CodeAlpaca [] with no attack, under attack and
with our defense in Table 6. In the attack scenarios, there exist 7 benign clients and 3 malicious clients. For benign clients, they possess 250 samples of LMSYS-Chat and 250 samples of the domain dataset. Malicious clients possess 500 samples of MaliciousGen from Mistral. For evaluation, we utilize HumanEval [] for coding task evaluation.

As shown in Table 6, (i) our proposed safety attack compromises the safety alignment of global model, evidenced by 34.62% decreases in MD-Judge score. (ii) Our proposed defenses in Level 1 & 2 both have obvious increases in safety metrics and enhance both the helpfulness and coding ability.

**Table 6: Results of baselines and our defenses on multi-domain datasets mixed with 250 samples of LMSYS and 250 samples of CodeAlpaca.**
| Evaluation Metric $\uparrow$ | Rule | MD-Judge | RM | MT-1 | HumanEval pass@1 |
| --- | --- | --- | --- | --- | --- |
| FedAvg (No Attack) | 60.00 | 42.12 | -2.15 | 4.08 | 17.07 |
| FedAvg [] | 35.19 | 7.50 | -3.77 | 3.86 | 14.63 |
| Krum [] | 39.42 | 12.12 | -3.51 | 4.13 | 17.68 |
| DnC [] | 39.04 | 11.73 | -3.71 | 4.41 | 18.29 |
| Ours: Level 1 | 55.96 | 25.77 | -2.94 | 4.50 | 15.24 |
| Ours: Level 2 | 76.73 | 87.88 | -0.79 | 4.11 | 17.68 |

### B.5 Effects of Number of Steps for Defense

For Level 3 defense, we change the training steps in [100, 200, 300, 400, 500] across four settings in Table 1 and Table 2. We show the model performance on MT-1 and MD Judge with 5 different training steps in Figure 5. We can note that (i) in Figure 5(a), training for 400 steps consistently obtains the highest MT-1 score across four settings, indicating the optimal 400 steps for Level 3 facilitates the helpfulness of global model. (ii) As shown in Figure 5(b), Our proposed post-hoc defense strategy demonstrably improves safety for all training steps and across the four settings. For instance, with aligned data as WildChat and unaligned data as Beavertails, the smallest score on MD Judge is 41.73%, 29.42% outperforms FedAvg under attack.
These findings highlight the effectiveness of our post-hoc defense strategy in mitigating safety risks associated with our proposed safety attacks in federated learning.

Figure: (a) MT-1
Refer to caption: extracted/5669388/figs/mt_num_defense_steps.png

### B.6 Impact of Generated Data on LLM Fine-Tuning and Defense

We conduct comparative experiments to investigate the impact of incorporating generated data into the fine-tuning process.
Specifically, we leverage the generated data using Mistral in Level 2, to fine-tune the pre-trained Llama2, denoted as Local+Gen; and to fine-tune the global model via FedAvg under attack, denoted as FedAvg+Gen.
Figure 6 depicts the scores for four evaluation metrics of normal local-training, Local+Gen, normal FedAvg and FedAvg+Gen. Results show that (i) generated data is not sufficient for helpfulness. Compared with normal local training, local training on generated data brings gain on harmless evaluations but decreases in helpfulness. (ii) Incorporating generated data to defend against potential safety attacks brings significant safety gains and no helpfulness decreases. Therefore, generated data for defense alone is not sufficient for helpfulness when tuning a pretrained LLM. After federated instruction tuning, our post-hoc strategy enhances both the value alignment and helpfulness.

Figure: Figure 6: Four metrics results of normal local-training, local-training with generated data in Level 2 defense, normal FedAvg and FedAvg with generated data in Level 2 defense.
Refer to caption: extracted/5669388/figs/bar_plot_pretrained.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. In the experimental setups, the paper claims that four benign datasets were considered. However, I have not found the experimental results of the Dromedary-verbose and Wizard-evol datasets.

2. This paper proposes a post-hoc defense to remedy the damage caused by attacks during FL. I wonder if existing centralized fine-tuning defenses [3] can mitigate the proposed attack? What is the innovation of this paper in the defense method?

3. I want to know the performance of the proposed attack on different proportions of malicious clients.

Reference

[3] Liu, Kang et al. Fine-pruning: Defending against backdooring attacks on deep neural networks. RAID’2018
soundness: 2
strengths: 1. This paper for the first time reveals the vulnerability of safety alignment of LLMs trained via federated instruction tuning.

2. This paper proposes the first data poisoning method that aims to compromise the safety of FedIT.

3. This paper proposes a post-hoc defense method that effectively enhances the safety alignment of compromised LLMs.
summary: The paper addresses the vulnerabilities in the safety alignment of large language models (LLMs) trained through federated instruction tuning (FedIT). It introduces a novel safety attack method that allows malicious clients to compromise the safety of LLMs by using unaligned datasets, which can be generated automatically without human intervention. Meanwhile, this paper proposes a post-hoc defense method that involves the server generating aligned data to fine-tune the LLMs, thereby improving their safety alignment. Extensive experimental results demonstrate the effectiveness of the proposed attack and defense methods.
weaknesses: 1. The baseline defense methods considered in this paper are all relatively outdated. It is better to consider the effect of the proposed attack under the defense method after 2022, such as [1][2].

2. This paper does not discuss whether the post-hoc defense will cause the performance degradation of the LLM model.

Reference

[1] Lu, Zhi et al. Split Aggregation: Lightweight Privacy-Preserving Federated Learning Resistant to Byzantine Attacks. TIFS’2024

[2] Wan, Wei et al. A Four-Pronged Defense Against Byzantine Attacks in Federated Learning. MM’2023

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Typically, applying defense mechanisms would lead to a reduction in accuracy. I'm curious if this also holds true for LMs. Since the approach involves using a different dataset for fine-tuning the aggregated model at the server, will this fine-tuning compromise the performance of the benign local models?
soundness: 3
strengths: 1. This paper addresses safety alignment in federated instruction tuning, a problem that is both important and interesting. The authors propose attacks along with corresponding defenses to against the attacks.

2. The paper introduces a post-hoc defense mechanism that uses generated safe data to finetune the aggregated model and restore safety alignment without inspecting individual client models.

3. The authors conducted extensive experiments across multiple datasets and defense scenarios. The results show that the proposed attack can significantly compromise the model's alignment, while the defense method can restore alignment effectively.
summary: This paper addresses attacks and defenses of safety alignment in federated instruction tuning. The authors propose a post-hoc defense method that fine-tunes the aggregated model with safe data at the server, thus restoring safety alignment without detecting malicious client models. Experiments demonstrate that the safety attack can significantly reduce the model’s safety rate, while the defense approach effectively restores the model's safety alignment.
weaknesses: 1. The baseline defense methods used for comparison may not be well-suited to the attack in this paper. Methods like Krum, Median, and Trimmed Mean are designed primarily for Byzantine attacks that manipulate model weights, rather than poisoning local training datasets. Also, methods such as Foolsgold can be effective against backdoor attacks that might poison training data, e.g., label flipping attacks, but these methods were developed for classification tasks. However, this paper addresses text-generation tasks with different goals and attack mechanisms. Thus, I'm concerned about the relevance and effectiveness of these baseline methods.

2. The paper mentioned using a single NVIDIA RTX 3090 GPU to train models across 10 clients, with 3 clients participating training in each round. However, it's not clear how a single RTX 3090 can support fine-tuning three LLaMA2-7B simultaneously. Also, are the benign local models are trained with IID data?

3. The motivation for training LLMs with FL is unclear. While FL is primarily used to protect the privacy of local data, LLMs are often trained on publicly available datasets, which may reduce the need for privacy-preserving approaches in this context. I understand there may be sensitive scenarios, such as in medicine, law, or finance, where each FL participant wants to protect their data. However, these cases would involve different data distributions across FL clients, which are different from the experiment setting.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Thank the authors for their work. Please see my questions as follows.

1. Could the authors clarify the rationale behind the attacker's objective of making LLMs behave harmfully only for harmful instructions? Have they considered scenarios where attackers aim to make LLMs behave harmfully for all types of instructions?

2. The authors assumed that 30% of users are malicious. The authors also stated for each round, only 3 out of 10 users were sampled. Is it randomly sampled? Moreover, for some rounds, users are all normal or malicious, which means this sampling process will naturally deal with malicious attacks. Could the authors provide a sensitivity analysis on the percentage of malicious users? Additionally, it would be valuable to see a breakdown of performance for rounds with different compositions of benign and malicious users to better understand the impact of the sampling process on the attack and defense effectiveness.



3. The authors performed most experiments on LLAMA-7B, and it is good to compare with other models. Could the authors provide hypotheses or analyses on why different models show varying levels of vulnerability to the proposed attack, as seen in Fig. 4? This could help in understanding if certain model architectures or training approaches are inherently more robust to such attacks.
soundness: 3
strengths: 1. The paper explores the safety topic in federated learning, which is important to understand especially in the era of LLMs.

2. The paper is well-written and the logical flow of the paper is clear and easy to follow.  

3. The paper considers the scalability of the proposed work.
summary: This paper reveals that the current form of FedIT presents vulnerability when malicious clients train their local models with harmful datasets, resulting in compromised safety alignment of LLMs. While existing defense methods fail to address such attacks, this paper proposes a post-hoc defense by fine-tuning central LLM with harmless instructions and responses that are generated by external LLMs without any human efforts. The evaluation demonstrates the effectiveness of the proposed attack and improves the safety alignment of the proposed defense.
weaknesses: 1. The authors should explain the assumption that malicious users’ objective is to make LLMs behave harmfully while given harmful instructions and behave normally when given normal instructions. 

2. The proposed attack is clear, yet the experiment setup (i.e., 30% are malicious users) is missing explanations.

3. The tradeoffs and benefits of the proposed defense method over existing defense methods need to be clarified. A clear comparison would have been helpful.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: N/A
soundness: 2
strengths: *S1.* This manuscript identifies vulnerabilities in FedIT caused by malicious data, which could compromise the alignment of large language models (LLMs).
summary: This manuscript  presents a safety attack against federated instruction tuning (FedIT) along with a corresponding defense method. The focus is on data-based attacks, where the attacker (one of the parties) can manipulate the training data while adhering to the training protocol, without directly modifying the model parameters. The attack is performed using simple instruction tuning with a malicious dataset (obtained from either public or generated data). To defend against this attack, the authors propose fine-tuning with aligned data after training. The experiments show the propose method outperforms some baselines.
weaknesses: *W1.* The attack model requires further clarification regarding the feasibility and necessity of data-based attacks. Typically, in FedIT, the attacker is assumed to be one of the parties who can control both the parameters transferred to the server and the training data. However, in this manuscript, the attacker is assumed to manipulate only the training data while leaving the local model parameters untouched. Additional explanation is needed on how this assumption aligns with real-world applications.

*W2.* The related work section lacks thorough comparisons. The manuscript claims to be the first work on data-based attacks, yet it is unclear when a data-based approach is preferable to a model-based one (see W1). Therefore, other attacks (both model-based and data-based) should be discussed and compared in the experiments.

*W3.* The experiments based on OpenFedLLM are not convincing. It is recommended to use more state-of-the-art or widely adopted LLMs for experimentation. Furthermore, stronger baselines, such as attacks that directly tune the parameters (see W1), should be included.

*W4.* There is limited technical challenge addressed in this manuscript. Simple fine-tuning with malicious or aligned data is applied without any specific techniques or novel approaches, providing limited insights for future research.

Meta-review: The paper addresses the vulnerabilities in the safety alignment of large language models (LLMs) trained through federated instruction tuning (FedIT). It introduces a novel safety attack method that allows malicious clients to compromise the safety of LLMs by using unaligned datasets, which can be generated automatically without human intervention.

+ The paper addresses an important problem.
+ The paper is well-written.
+ The experiments are extensive.

- Some of the baselines are not suitable.
- Both the attack and the mitigation strategy appear not to be specific to federated learning.

Decision: Accept (Poster)