SABRE-FL: Selective and Accurate Backdoor Rejection for Federated Prompt Learning
Authors: Momin Ahmad Khan
Yasra Chandio
Fatima M. Anwar
Venue: ICLR 2026 Poster (2026)
Keywords: Federated Learning
Poisoning Attacks
Multimodal Learning
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Federated Prompt Learning has emerged as a communication-efficient and privacy-preserving paradigm for adapting large vision-language models like CLIP across decentralized clients. However, the security implications of this setup remain underexplored. In this work, we present the first study of backdoor attacks in Federated Prompt Learning. We show that when malicious clients inject visually imperceptible, learnable noise triggers into input images, the global prompt learner becomes vulnerable to targeted misclassification while still maintaining high accuracy on clean inputs. Motivated by this vulnerability, we propose SABRE-FL, a lightweight, modular defense that filters poisoned prompt updates using an embedding-space anomaly detector trained offline on out-of-distribution data. SABRE-FL requires no access to raw client data or labels and generalizes across diverse datasets. We show, both theoretically and empirically, that malicious clients can be reliably identified and filtered using an embedding-based detector. Across five diverse datasets and four baseline defenses, SABRE-FL outperforms all baselines by significantly reducing backdoor accuracy while preserving clean accuracy, demonstrating strong empirical performance and underscoring the need for robust prompt learning in future federated systems.

SABRE-FL: SELECTIVE AND ACCURATE BACKDOOR REJECTION FOR FEDERATED PROMPT LEARNING
Momin Ahmad Khan, Yasra Chandio & Fatima Muhammad Anwar University of Massachusetts Amherst (makhan, ychandio fa fanwar @umass. edu
ABSTRACT
Federated Prompt Learning has emerged as communication-effiiies and privacy- preserving paradigm for adapting large vision-language models like CLIP across decentralized clients. However, the security implications of this setup remain underexplored. In this work, we present the first study of backdoor attacks in Federated Prompt Learning. We show that when malicious clients inject visu- ally imperceptible, learnable noise triggers into input images, the global prompt learner becomes vulnerable to targeted misclassification while still maintaining high accuracy on clean inputs. Motivated by this vulnerability, we propose SABRE- F a lightweight, modular defense that filters poisoned prompt updates using an embedding-spaae -spann anomaly detector trained offline on out-of-distribution data. SABRE-FL requires no access to raw client data or labels and generalizes across diverse datasets. We show, both theoretically and empirically, that malicious clients can be reliably identified and filtered using an embedding- based detector. Across five diverse datasets and four baseline defenses, SABRE-FL outperforms all base- lines by significantly reducing backdoor accuracy while preserving clean accuracy, demonstrating strong empirical performance and underscoring the need for robust prompt learning in future federated systems.
INTRODUCTION
Federated Learning McMahan al. (2017) enables decentralized model training across multiple users while keeping data local, thereby preserving privacy and reducing centralized risks. In FL, clients independently train models on local data and share only model updates with server, which aggregates them into a global model. Due to its settings like Google's 's Gboard gbo (2017) for next-word word prediction, Apple's Siri tec for automatic speech recognition, and WeBank for credit risk prediction webankcredit, Recent advances have extended FL to support more expressive models, such as vision-languee models, by integrating prompt-based learning Zhou et al. (202bb); Guo et al. (2023b) Deng et al. (2024).
prreservinggg nature, FL has been adopted in
Prompt learning a recent paradigm that adapts large pre-trained models such as OpenAl's CLIP (Contrastive Language-Imape Pretraining) Radford et al. (2021) to downstream tasks by optimizing lightweight, learnable input prompts instead of finetuning the full model. Originally developed cen- tralized settings, prompt learning has shown impressive few-shot generalization. task transferability, and reduced compute cost, particularly with vision-language models /hou et al. (20226 a). Motivated by these advantages, recent works have introduced prompt learning into FL al. (2023b): [Weng iet al. (2024), giving rise to federated prompt learning (FPL). In FPL. clients independently optimize prompt vectors while keeping the model backbone frozen, and share only these prompts with the server. This design greatly reduces communication and memory overhead and enables efficient cross-client adaptation multimodal and heterogeneous environments.
Despite its appeal. FL is not inherently secure Kairouz et al. (2019). In practice, some clients may behave maliciously, either by corrupting their local training data or manipulating model updates, to influence the behavior of the global model Fang et al. (2020); (Shejwalkar et al. (2022); Wang ct al. 2020 Shejwalkar & Houmansadr (2021); Baruch et al. (2019); Mhamdi et al. (2018 8 Xic
We will release the open source code with the final version of this paper.

---

elephent
flamingo
Golphin
enooder
Text features
CLASS
Text Encoder
Learnable
prompts
Context
generator
Actual Label elaphent
Image
Encoder
Prodicted
nolise trigger
Learmable
feature
Image
calculation
Siilianty
colphin
label
Figure I: Backdoor attack on coompt-earning-baseee multimodal models. A learnable and impercep- tible noise trigger is added to the image that results in a poisoned image embedding, which is then used generate the learnable prompts. This addition of noise causes the image features to deviate from their respective text features in the embedding space, thereby causing misclassification.
cet al. (2019b); Bhagoji et al. (2019); Bagdasaryan et al. (2020a). A particularly insidious example is the backdoor attack Nguyen et al. (023); Zhang et al. (2023); ; Bagdasaryan al. (2020b); Wang
et al.] (2020). in which an adversary injects carefully crafted inputs (called triggers) into local training data (Figure Tp. These triggers cause the global model misclassify specific test-time inputs while preserving high accuracy on benign samples. Prior work on backdoor attacks in FL has largely focused on traditional classification tasks in unimodal settings Nguyen et al. (2023); Zhang al. (2023), leaving the security properties of multimodal and prompt-based FL systems underexplored.
While prompt learning in FL is gaining momentum, its security properties remain largely unexamined. This raises a key question: how vulnerable is Federated Prompt Learning to backdoor attacks? In this work, we show that prompt learners in FL are highly susceptible to backdoors, even when model updates are limited to prompt vectors. We introduce a backdoor attack that inserts a learnable, visually imperceptible trigger into a subset of clients' training data. The attack draws inspiration from BadClip Bai et al. (2024) who design a trigger-based backdoor attack for prompt-learning in the centralized setting. In our attack each malicious client has its own malicious trigger that pushes the prompt embeddings toward target label in CLIP's semantic space, leading to high- confidence misclassification at inference. The attack remains stealthy and retains high clean accuracy across clients, matching performance observed in centralized prompt tuning. This demonstrates that Federated Prompt I Learning is vulnerable to trigger-based based backdoor attacks even when a few clients act maliciously. To the best of our knowledge, we the first study backdoor attacks in this setting, i.e., trigger-based attacks in multimodal federated prompt learning.
Motivated by this, we design SABRE-FL (Selective and Accurate Backdoor Rejection), lightweight server-side defense tailored for prompt-based FL. Our key insight is that backdoored prompt vectors yield representations that deviate from the distribution of clean data in CLIP's embedding space. SABRE-FL trains a detector offline, on an out-of out-of-distributee diddstriibution dataset, to recognize these deviations. Importantly, the detector does not require access to client data, labels, or downstream tasks. By leveraging this separation in representation space, SLBRE-FL. identifies and filters poisoned updates with high precision, maintaining clean model performance while eliminating backdoor impact.
Contributions: In our work, we address the critical issue of backdoor attacks in federated prompt learning. In doing so, we make the following key contributions:
We introduce the first backdoor attack specifically targeting prompt learning in FL ($3) The attack injects a visually imperceptible, learnable noise trigger that is optimized to shift prompt representations toward target class semantically. The attack achieves high backdoor success while preserving clean accuracy, and remains effective even when only a small fraction of clients are compromised, revealing a vulnerability in prompt-based FL systems.
Designing SABRE-FL: We propose SABRE-FU FL ($4). a lightweight, generalizable defense framework that detects poisoned prompt updates at the server using a classifier trained on out-of-distributio embeddings. We formalize its representation-space decision boundary and provide theoretical conditions for generalization.
Comprehensive evaluation and analysis across five datasets and four defenses; Trimmed Mean, Median, Norm Bounding, and FLAME ($5.1), shows that SABRE-FL consistently outperforms existing methods by achieving lowest backdoor accuracy while maintaining clean accuracy. t-SNE plots and ablations ($5.3) confirm its generalization and effectiveness under diverse FI. and prompt learning configurations.
2

---

2 BACKGROUND AND RELATED WORK
2.1 FEDERATED LEARNING (FL)
In FL Kairouz et 2019); McMahan al. 2017 a central entity, known the server, aims train a global model, 89, using private data distributed across multiple clients, without directly accessing their data. In each communication round, the server selects them the current global model g' where denotes the round index. Each selected client & computes an update V using its local dataset DR, and returns it to the server, which aggregates all updates using a predefined aggregation rule, such as FedAvg McMahan et al. (2017).
of N available clients and sends
In FedArg, a client k fine-tunes fr on their local data using stochastic gradient descent (SGD) for fixed number of local epochs E, resulting in updated local model of. The client then computes their update as the difference V = of 0g and shares V with the server. Next, the server computes an aggregate of client updates, fags using mean, i.e., Vigge" = Smean (V (KE(())) and updates the global model of the (t + 1)th round using SGD and server learning 1) as: 0,+1 03 + 1/
2.2 PROMPT LEARNING WITH VISION-LANGUAGE MODELS
Vision-Language Models:
Large vision- languaee models (VLMs), such as CLIP Radford et al. (20II). have demonstrated remarkable generalization across diverse downstream tasks. By aligning images and text in a shared semantic space, these models enable strong zero-shot and few-shot performance without task-specific supervision. However, their size, often exceeding hundreds of millions of parameters, makes traditional fine-tuning computationally expensive and bandwidth- intensive, particularly in distributed or resource-constrainee environments.
Prompt Learning Zhou et al., (2022b): Prompt learning adapts large pre-trained models to down- stream tasks by introducing set of learnable prompt vectors that are prepended to the model input. During training, only these prompts updated, allowing efficient task adaptation while keeping the backbonn frozen. This reduces the number of trainable parameters and computational cost, making the approach particularly attractive for few-shot and assource-consraingg Prompt learning has been shown to be effective across multiple modalities Khattak et al. (2022); Zhou et al. b). In CLIP--bsed architectures, this involves optimizing a set of context vec-
settings.
tors V == U1, ......, uy] E RNxe, where each v, is a learnable token embedding and e is the embedding dimension. Given an input image a and a class name embedding C, the image encoder f(x) and the text encoder o (V, c.)) produce modality-aligned representations. The prediction probability computed using cosine similarity:
exp(sim(f(=),otiVvoco c.Hl/r)
ply y = w)
(1)
ΣØ=1 exp(sim(f(())) 9((V...)))))
where 7 a temperature parameter and sim(.,) denotes cosine similarity.
Prompt Learning in FL: The benefits of prompt learning mentioned above have motivated its integration into the federated setting Guo al. (023b]a> a): Zhao al. (2023). In federated prompt learning, each client optimizes a local prompt vector while keeping the foundation model, e.g.. CLIP. frozen, and transmits only the prompt the server for aggregation. This substantially reduces memory usage and communication cost, making it feasible to deploy foundation models like CLIP in privacy-po preeeeen,,,,
moaddiidt-lliiitee environments. Such systems have demonstrated strong downstream performance across vision and multimodal tasks while maintaining FL's privacy and scalability benefits.
Despite these advantages, the security implications of prompt learning in FL remain largely un- explored. In particular, unclear whether prompt learners, given their limited parameter space and semantic alignment with frozen backbones, susceptible to adversarial manipulation, such as backdoor attacks. This presents a critical and underexplored vulnerability in the growing area of federated foundation model adaptation.
2.3 BACKDOOR ATTACKS
Backdoor attacks Bai et al. (001 2022 2023a): Gao et al. (003); ;; Li et al. (2021): Turner et al.
(009 Xu et al. (2024) Ya al. (2024)
class of training-time data poisoning techniques
3

---

update geompt Inarner with locst data
clienta send local prompt naamnr sener
Text Encoder
Scores
Text Festtures
Local update rule
Laarmebla Prommpt
peneattrr
server updates the global prompi lesr
Image
image
Nosin
serves sunda pr prompt inarner to clients
Figure 2: An overview the attack in an FL setting. A malicious client embeds a learnable noise trigger into images. The context generator helps optimize the prompts according to the image features.
wherein an adversary injects carefully crafted samples into the training set with the goal of inducing targeted misbehavior at test time Bagdasaryan et al. (2020a); Hanif et al. (2024). These poisoned samples contain imperceptible or benign-looking trigger, such as small patch in the input, and are assigned a target label of the attacker's choosing. Once trained on such data, the compromised clean inputs but misclassifies any input containing the trigger to the
model behaves normally target class. This selective misbehavior makes backdoor attacks particularly insidious, as they are challenging to detect using standard validation procedures. Backdoor attacks have been studied across multiple modalities including vision Gu et al. (017). language Kurita et al. (2020), and multimodal models Liang al. (2024); ; Bai al. (2024) and have proven effective even
settings like FL. where model updates rather than raw data are shared.
Backdoor Attacks in Federated Learning: Backdoor attacks pose a serious security threat to FL, allowing adversaries to embed malicious behavior into the global model by manipulating small number of clients during training [Bagdasaryan et al. (2020b); WWag et al. (2020): (Shejwalkar & Houmansadr (2021). These attacks typically preserve high accuracy clean inputs while causing targeted misclassioe iisiiassiiiiotoonn on inputs containing an attacker-defined trigger. Early approaches rely on fixed triggers Sun et al. (2019); Xie et al. (2019a); BBhagoji et al. (2019). while more recent methods optimize trigger patterns to maximize attack success and stcalthiness Nguyen et al. (203)) y Zhang et al.
(2023). For example, A3FL Zhang et al. (2023) predicts the movement of the global model updates and improves attack durability by ensuring the backdoor persists across global aggregation rounds. Similarly, IBA Nguyenn et al. (2023) jointly optimizes visually stealthy trigger and selectively poisons models' parameters that are less likely to be updated by the main task's 's learning process, achieving durable and stealthy backdoor effect.
3 BACKDOOR ATTACKS ON PROMPT LEARNING IN FL
3.1 OVERVIEW
While backdoor attacks have been extensively studied in traditional unimodal FL settings, their feasibility in multimodal federated prompt learning remains underexplored. Unlike traditional full- model FL. it exposes narrower attack surface, limited prompt vectors, raising questions about strength, persistence, and stealth of such attacks. These differences motivate our central hypothesis.
Hypothesis. We hypothesize that backdoor attacks capable of degrading centralized prompt learning can similarly succeed in federated prompt learning. Despite the distributed setup and aggregation dynamics, prompt-based FL remains vulnerable targeted manipulation, allowing adversaries to induce misclassifications on trigger inputs while preserving overall model utility on clean data.
Positioning our work relative to existing literature, Several recent works have demonstrated the vulnerability of contrastive vision- assion-language models like CLIP to backdoor attacks in centralized settings. Notably, BadCLIP/Bai al. (2024) introduces a powerful trigger-aware attack that jointly manipulates both the image and text encoders using prompt-conditioned triggers. A similar vari- ant [Liang et al. (2024) improves stealth and robustness using dual-embedding alignment. Other works such as BadEncoder Jia et al. (2022) and contrastive poisoning attacks Carlini & Terzis (2021) inject backdoors directly into frozen image encoders or pretraining datasets. While effective, these attacks are designed for centralized pretraining regimes. In this paper, focus exclusively on the BadCLIP attack due to its compatibility with prompt tuning and its relevance to the downstream FL scenario explored in work.
4

---

Theoretical Motivation, In CLIP-basee based prompt learning Zhou et al. (2022b a). classification is based on the cosine similarity between an image embedding f()) and a prompt-conditions text embedding ((V,, c:)) for class i. To induce targeted misclassification toward specific class it suffices to craft an input x² such that:
sim(f(("), I(V, cx)))) ((
Vy Ht
(2)
This condition ensures that the model classifies x" as belonging to the target class t. In practice, our attack injects a visually imperceptible trigger, as shown in Figure [T] into local training data and optimizes it to shift image embeddings toward g((V. ..t)), effectively planting a backdoor in the global prompt learner. While this idea inspired by prior work on backdoor optimization [Bai et al. (2024); ZZang et al. 2023); Nguyen et al. (203)), adapting it to the prompt-only FL. setting introduces new challenges: the global model is now updated solely via lightweight prompt vectors, and the image encoder remains frozen. This means the backdoor signal must propagate indirectly through prompt aggregation, requiring the trigger to consistently bias prompt updates without direct influence over model weights, making the optimization problem both weaker in signal and more sensitive to noise. The attack is visually explained in Figure !
Evaluation Metrics. We report two metries: Clean Accuracy (CA) and Backdoor Accuracy (BA). Let Detemm = ((x., }i)} denote the clean test set and Dha = {(xt,)) the backdoored test set, where a. 24 @ is the triggered input for target label y/-- Clean Accuracy, the percentage of clean inputs predicted correctly, is defined CA |Deeal [x)) = We]. while Backdoor Accuracy, the percentage of backdoored inputs predicted as the target label, is BA
Deal (0((()) = yel.
3.2 THREAT MODEL
Objective. The adversary's goal is to perform targeted backdoor attack in a federated prompt learning setup. By injecting a learnable, visually imperceptible trigger into a subset of training inputs at compromised clients and relabeling them to fixed target class, the attacker aims to corrupt the global prompt learner. At inference time, inputs stamped with the trigger are misclassified as the attacker's chosen class, while clean inputs remain unaffected, thus maintaining high clean accuracy.
Capabilities. We assume a standard FL setup with N clients and a central server aggregating client prompt updates. The adversary controls a fraction m/N of clients, 25% by default, consistent with prior works Cao et al. (2023 202T). The attacker can:
Modify subset of local training data by adding learnable backdoor trigger to inputs.
Relabel triggered samples to the desired target class, known in literature as dirty-label attack She
jwalkar et al. 2023); : Gu et al. (2019); Chen et al. (2017): Sarkar al. (2021); ZZen al. 2021
Nguyen & Tran (2020): Li et al. (2021).
Optimize the trigger jointly with the prompt learner at each malicious client to maximize its effect on the global prompt vector.
Knowledge. Since the attacker controls client devices, it naturally has access to the full prompt learning setup, including model architecture, frozen CLIP backbone, and training procedure. This is standard assumption in federated backdoor attack literature Bagdasaryan et al. (2020b); (Shejwalkar et al. 2022), and reflects realistic adversaries in open open-source or distributed deployments where models like CLIP are publicly available Radford et al. 2021).
3.3 DESIGN OF THE BACKDOOR ATTACK IN AN FL SETTING
We illustrate the overall system of the backdoor attack in Figure 2 At the beginning of each commu- nication round, the server distributes (step 1) the current global prompt learner all participating clients. Unlike traditional FL systems that transmit full model parameters, prompt-based FL transmits only the learnable prompt vectors, significantly reducing communication overhead. The clients keep their model backbones, the image encoder fing and the text encoder fiet. frozen. During local training (step 2). each client fine-tunes the received prompt vectors on its private data. Malicious clients, however, inject a learnable additive noise trigger into subset of their training images and assign these poisoned samples to an attacker-specifie target label, Mtarget- The objective of malicious clients is to optimize their prompt learners such that the presence of the trigger at inference time reliably
5

---

causes misclassification, without noticeably affecting clean accuracy. After completing local updates, clients send their locally adapted prompt vectors back to the server (step 3). The server aggregates (step 4) these updates to form the new global prompt learner, which is then redistributed to all clients. This process repeats over multiple rounds until convergence.
Attack Formalization: Let (a, y) be clean image and label pair, with X and y E y Let fimg : X' - Rd be the image encoder and fieat y + RRd be the text encoder from a frozen CLIP model. Prediction is defined as:
y = arg max cos (z)) Fieu .EY
The attacker injects a learnable trigger E X such that x* t indistinguishable from æ in pixel space, but shifts its embedding in CLIP space.
Goal:
cos(Simg(2") (2*) Seext (Vhurges) > cos(Simg (x*), (()
This causes the model to predict Marget instead of the true label y. The trigger ł is learned via gradient descent to consistently shift embeddings toward fiex: across poisoned samples.
3.4 ATTACK IMPACT
Table 1:
Accuracy with no attack (clean model),
clean inputs under attack, and backdoored inputs.
We now assess the effectiveness of the back- door attack in standard federated prompt- learning setup, where 25% of clients are ma- licious. These clients inject learnable noise trigger into subset of their local data and rela- bel the triggered samples to a fixed target class. The goal is induce targeted misclassifications on trigger-inserted test samples, while preserv- ing high performance clean data.
Dataset
No-Attack
Clean
Backdoor
Flowers Pets
80.9
77.9
41.7
94.5
94.2
16.3
DTD
65.2
65.6
34.8
Aireraft
32.3
32.8
93.9
Food101
90.7
90.0
20.6
Backdoor Effectiveness: Table [ Figure show the results of the attack across five datasets. Refer to Appendix for setup details. We observe that the global model maintains high clean accuracy on all datasets, indicating that benign generalization is largely preserved. At the same time, the backdoor accuracy which is defined as the fraction of trigger-inserted test samples classified as the attacker's 's target label is significantly elevated, particularly for datasets like FGVC Aircraft (93.9%) and Flowers (41.7%). These results confirm that Federated Prompt Learning systems are vulnerable to backdoor injection even under strong aggregation, and that malicious clients can effectively implant targeted behaviors without degrading global model performance on clean data.
Comparison with Centralized Backdoor Attacks: We compare our FL backdoor attack against its centralized counterpart, FIPCLIIP Bai et al. (2024), which serves as the bascline for prompt-lcarning backdoor attacks non- federated settings. BadCLIP achieves near 100% backdoor success by directly poisoning a large portion of the train- ing data and optimizing the trigger in a fully centralized regime. In contrast, our setting uses the standard FedAvg aggregation algorithm and models more realistic adver- sary: only a small subset of clients are malicious, and poisoning is confined local updates. This naturally di- lutes the backdoor signal during aggregation and results in lower backdoor accuracy compared to the centralized case. Despite this, our attack achieves high success rates on several datasets, demonstrating that prompt-based FL remains vulnerable even with limited adversarial partici- pation. In Table we report results under the no-defense occur with
Rowers
Alcot Food
Pets
370
10
(a) Clean Accuracy
110
Pats
Food
40-
40
scenario highlight how much damage the default FedAvg setup. We analyze the effectiveness of standard defenses in mitigating this attack later in $5.1
10
(b) Backdoor Accuracy Figure Accuracy during attack.

---

4 SABRE-FL: SELECTIVE AND ACCURATE BACKDOOR REJECTION FOR FEDERATED PROMPT LEARNING
Having demonstrated the vulnerability of federated prompt learning to targeted backdoor attacks, we now propose Selective and Accurate Backdoor REjection for Federated Prompt Learning (SABRE- FL), a lightweight defense that detects and filters poisoned client updates at the server.
Our key insight is that backdoored inputs induce systematic shifts in the learned representations, as visualized later in $5.2 Even when the trigger is visually imperceptible, it alters the image embedding in a consistent direction enough to cause the downstream model misclassify the input. This deviation acts as a double-edged sword: it is the very signal that enables the attack, but also the very signal we exploit to build our defense. A similar observation was made BadCLIP Bai et al. (2024), which showed that the success of prompt-based backdoors arises from consistent embedding- drift toward the target class. We ask question: If this embedding deviation is strong enough to fool the downstream classification model, can it not also be used detect that the input has been poisoned?
Core idea. Rather than detecting poisoning in pixel parameter space, operate in the embedding space where poisoned examples exhibit a consistent statistical signature. By training a binary classifier on clean and triggered embeddings in an auxiliary setting, we learn to detect this signature.
4.1 FORMALIZATION
Algorithm 1 SABRE-FL
1: Pre-train Detector:
Let fime(-) denote the CLIP image encoder. Given a clean input x, let z = fmg(( (x), and for its backdoored version x²
Generate Dan {(2, v)} from clean/poisoned
data
@ let 2 = fimg(2 (x*). Our defense relies on the as- sumption that backdoored embeddings exhibit a separation margin from the embeddings of the clean ones:
Train D: D= + {0. 1] using cross-entropy
2:
for each FL round L = to T do
3: Server sends prompt p--1 all clients
4:
for cach client C₃ do malicious then
5:
6:
Poison subset: x" = O Relabel x - cs train poisoned
| 22 > for some e >> 0 (5)
7:
data
We simulate this behavior by generating a train- ing dataset Damx= {(=;, ys) ) of clean and poisoned embeddings on an auxiliary dataset
8:
clse
9:
Train pk on clean data
10:
end if
11:
Send pæ. embeddings (zfi} to server end for
(Caltech-101). Here, y4 E {0, 1) indicates
whether 2; is clean or poisoned. We train a detec-
13:
for each Cs
tor D Rd + {0, 1) by minimizing standard
14:
Compute S₁ = mn E, D(=))
binary loss:
Remove top-m clients with highest Sy. end for
16:
N
17: Aggregate accepted {pk} -
min
((D(z)0).u) v.)
(6)
18:
end for
Inference Rule. At inference time, when a client Cs submits set of embeddings { we compute the mean detector score:
1 Σ D( 71
S
Rather than using a fixed threshold т, we adopt a rank-based heuristic: each round, the m clients with the highest number of flagged embeddings are excluded from aggregation. This approach assumes an upper bound on the number of malicious clients, consistent with prior work Yin et al.
(2018); Shejwalkar & Houmansadr (2021) |Fang al. (2020); Cao et al. 2023): Zhang et al. (202)) More details on client filtering are in Appendix B
Lemma. If consistent margin exists and D achieves zero near-zero training on Daux. then D is expected to generalize well to unseen clients using a noise trigger. This reflects the distributional stability of backdoored embeddings under the frozen encoder.
7

---

Table 2: Clean and backdoor accuracy on five datasets. Best backdoor accuracyylowest) is bold.
Flowers
Pets
DTD
FGVC Aircraft
Food101
Defense
Clean
BD
Clean
BD
Clean
BD
Clean
BD
Clean
BD
No Defense
77.9
41.7
94.2
16.3
65.6
34.8
32.8
93.9
90.0
20.6
Trmmmnd Mein
76.8
12.3
93.7
5.6
63.7
31.0
32.4
83.1
90.0
6.4
Median
77.4
10.4
94.1
5.3
65.9
28.1
32.1
79.4
90.1
5.5
Norm Bounding
79.0
22.0
92.6
22.5
67.6
37.5
30.9
86.2
89.7
17.2
FLAME
3.8
7.8
66.0
31.5
SABRE-FL FL (Curs)
76.6
1.1
94.5
4.4
64.9
6.8
32.1
7.6
90.6
1.9
4.2 DETECTOR TRAINING AND DEPLOYMENT
To operationalize the formalization of our defense, we construct an auxiliary training dataset Daux = {(z,, D:)};=1 composed of CLIP image embeddings and binary labels indicating whether the embedding originates from clean or poisoned input. To simulate this, we use Caltech-101 as held-out auxiliary dataset and apply our trigger injection method (Algorithm [ line 6). to subset of images to produce poisoned samples. Both clean and triggered images are passed through the frozen image encoder fing(.) and fixed prompt learner to obtain their embeddings. These embeddings then labeled as clean (y4: = 0) or poisoned (y)= = 1) to construct the training set. We defer the rest of the training details to Appendix C.4
4.3 PRIVACY CONSIDERATIONS
SABRE-FL operates solely in the embedding space and does not require access to raw data, labels, or gradients. Clients share only CLIP-encode image representations with the server which are compressed, aask-agnostic vectors produced by a frozen backbone. This strategy is consistent with prior FL paradigms such vertical FL Liu et al. (2024); Fu et al. (2022); Bai et al. (2023b) and split learning Vepakomma et al. (2018; Thapa et al. (2022), where intermediate features are shared across parties. Moreover, since
frozen encoder, the embeddings are less likely to leak private information (more details in Appendix (A44). Unlike gradients or label-conditioned outputs, CLIP embeddings not trained to retain input-specific details reconstruct original data. We acknowledge that data extraction attacks are an evolving research concern Carlini et al. [Bale] et al. ) however, our approach avoids sharing data, labels, gradients, components that are more strongly correlated with reconstruction leakage.
EXPERIMENTS AND RESULTS
5.1
RESULTS
Due to space constraints, defer setup details and additional experiments to Appendices C
Effectiveness SABRE-FL. We compare our proposed defense, SABRE FL, to four widely-used robust aggregation techniques: Trimmed Mean Yin el al. (2078); Xie et al. (2018), Coordinate-wise Median Yin et al. (2018), Norm Bounding Sun et al. (2019), and FLAME el al. (202)). Results across five datasets are shown in Table 2 Our defense achieves the best backdoor mitigation across all datasets, consistently outperforming all baselines. Notably, SABRE-FL reduces backdoor accuracy to near zero (as low 1.1% on Flowers and 1.9% on Food101) without degrading clean accuracy. In fact, clean performance remains comparable or superior to baseline methods, highlighting that aggressive filtering of poisoned clients does not impair generalization. While existing methods do reduce the backdoor accuracy relative to the no-defense baseline, they often leave a significant portion of poisoned influence intact, especially on challenging datasets like FGVC Aireraft and DTTD For example, FLAME achieves 16.4% BA on FGVC Aircraft, and Norm Bounding exceeds 30% BA on multiple datasets.
Robustness and Generalization. SABRE-FL operates without access to client data distributions or downstream task labels. The detector is trained once on Caltech- -00c and generalizes across diverse datasets in our evaluation (c.g., Flowers, DTD, FGVC Aireraft, Food101, Pets). This generalization holds across input domains such as fine- amne-rrined object categories (Flowers, Aircraft) and texture- based recognition tasks (DTD), as well across classification objectives ranging from animal species
8

---

Table 3: Backdoor attack effectiveness with and without SABRE-FL at 32-client scale, Each cell shows Clean Accuracy / Backdoor Accuracy (%).
Dataset
Flowers
Pets
DID
FGVC Aireraft
Food101
No Defense
74.9/43.5
88.8/
59.3 46.8
29.9/ 89.9
89.2/ 32.2
SABRE.FL
75.0/ 88.5
91.1
61.0/ 14.1
29.7/ 24.7
89.7 /28
(Pets) food recognition (Food101). Because the embedding deviation arises from the backdoor mechanism itself, not the specific data distribution, SABRE-FL reliably detects poisoning via consistent statistical signature in the embedding space. This highlights its robustness across both domains and tasks, making it broadly applicable in real-world federated deployments.
5.2 QUALITATIVE ANALYSIS
Secidand
To demonstrate why our detection mechanism works, we visualize the embeddings of clean images and their poisoned counterparts. The idea behind this experiment is noise is imperceptible in the visual space the human naked eye, but is it imperceptible in the embeddiin space to the model? This answered by visualizing the embeddings in low-dimensional space using technique like T-SNE Van der Maaten & Hinton (2008). In Figure 4] we show the T- SNE plots for Caltech-101. 101. We show a similar plot in Appendix [D] for Oxford Flowers. We first train model with backdoors using the technique similar to BadClip Bai et al. (2024), then we pass clean and noisy images through the image encoder and store the output embeddings. When we plot these embeddings using T-SNE, we see that there is clear divide between the features of the clean images and the backdoored images. This validates our intuition behind designing our defense, which lies in the simple fact that if the noise can be used to fool the model into predicting wrong class, that same noise can also be used to detect if an embedding comes from a clean image or poisoned one.
Figure 4: t-SNE visualization on Caltech embeddings. Clean and backdoored samples are clearly separable in the CLIP embedding space.
5.3 ABLATION STUDY
Cetorac)
Sackddor Dateras)
Impact of Prompt Shot Count:
Accurszyy (S))
The number of shots in prompt learning deter- mines how many samples per class are used to tune the prompt. We study how prompt strength affects both attack success and defense robust- ness via an ablation over 2, 4, 8, and 16 shots. For each setting, we report clean and backdoor accuracy, with and without our defense, across five benchmark datasets. Figure 5 shows results for DTD; remaining plots are in Appendix D.2
Figure 5: Varying number of shots for DTD
Without any defense, backdoor accuracy increases significantly as the number of shots grows, most notably in datasets like FGVC Aircraft and Food101, where attack success reaches over 85% at 16 shots. This trend suggests that prompt learners become increasingly susceptible to backdoor attacks they receive (more details in Appendix A.2). At the same time, clean accuracy also improves, reflecting the natural benefits of more labeled data. With our defense SABRE-FL enabled, however, backdoor accuracy remains consistently low (under 5%) across all shot counts and datasets. This indicates that our embedding- Crucially, clean accuracy under our defense matches or exceeds the no-defense baseline, confirming that the defense does not suppress benign updates. Overall, this experiment highlights that our method provides strong backdoor mitigation without compromising clean performance, even as model capacity increases with additional prompt shots.
supervision, likely due to stronger memorization of poisoned training samples
detector remains effective even as prompt learners become more expressive.

---

Robustness to Client Scaling: To evaluate the robust- ness of SABRE-FL. under increased scale, we replicate our backdoor attack and defense experiments with 32 clients. As shown in Table [] backdoor success rates rise sub- stantially in the absence of defense, reaching 89.9% FGVC Aircraft and 46.8% on DTD. When SABRE-FL. is enabled, backdoor accuracy drops to 24.7% and 14.1% %,
80
70
BTD
PSXC Aecrett
respectively, demonstrating that our detector remains ef- fective even as the number of participating clients grows. Clean accuracy also remains stable across all datasets, con- firming that the defense generalizes to larger federated populations without degrading utility.
50
100
(8)
(a)
Clean accuracy vs. malicious clients.
100-
00-
Effect of Malicious Client Proportion: We vary the proportion of malicious clients on both clean accuracy and backdoor success. In Figure 6 backdoor accuracy rises sharply the attacker fraction increases. At at-
Backdioor Accuracy
tacker rate of 25% the attack achieves 93.9% success on FGVC and 41.7% on Flowers. At 50% malicious more, backdoor accuracy exceeds 80% on most datasets and approaches 100% the highest setting. These results highlight the sensitivity of prompt-based FL to even adver- sarial participation, especially in few-shot regimes where each client contributes limited data. Notably, clean accu- racy remains largely unaffected across all configurations, indicating that the poisoned updates are stealthy and do not visibly degrade global model performance.
20
DTD
HOVC Arcort
25
50
100
Malicious Clients (%)
(b) Backdoor accuracy vs. malicious clients.
Figure 6: Effect of increasing malicious client percentage on model performance. Clean accuracy remains stable, while backdoor success increases sharply with more adversarial control.
LIMITATIONS AND FUTURE WORK
This work brings together three major research areas: federated learning, prompt learning, and backdoor attacks under a unified evaluation framework. Given the breadth of this integration, is naturally beyond the scope of single paper exhaustively explore all possible combinations of settings, attack strategies, and defense variants within this space. Our goal in this paper was to highlight critical and previously unexamined vulnerability: the susceptibility of federated prompt learning to targeted backdoor attacks. To that end, we carefully selected evaluation settings that isolate this problem and clearly demonstrate both the threat and the effectiveness of SABRE-FL.
However, there are limitations. First, while we focused on data poisoning attacks with learnable triggers, we did not explore model poisoning attacks Fang et al. (2020); (Shejwalkar & Houmansadr (202T), where the attacker perturbs client model parameters directly. Future work could compare the relative potency and stealth of model vs. data poisoning in prompt-based FL. Secondly, we used the CLIP ViT-B/16 backbone throughout this study; while it is representative and widely adopted model, future work may examine other vision- languaee backbones (c.g., ViT-L, EVA-CLIP, or OpenCLIP variants) to assess generalization across model families.
7 CONCLUSION
We show that backdoor attacks are a potent threat to federated prompt learning. We show their success, and use that to design robust defense, SABRE-FL, against such noise-trigger-bue attacks. Our defense is based on the core intuition that the backdoor noise trigger propagates to the embeddings as well. SABRE-FL a detector model that able to filter clean and noisy embeddings. Evaluation across five datasets and four baseline defenses shows that our defense outperforms all baselines.
8 ACKNOWLEDGEMENTS
We thank the ICLR reviewers for the insightful feedback. This work was supported by NSF award 2452819.
10

---

REFERENCES
How
Apple
personalizes Siri
without
hoovering
up
your
data.
https: www /ttp://wccccollyyyeeeeeee//////a
apple ai pwwsonalizes siri orrrsoaaaliaessssatreeeeeeetteed Learniiig
Federated learning: Collaborative machine machiee learning without centralized training data, https:
ai htt.:o///i....oo////////////eeee
html
2017.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In AISTATS, 2020a.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International conference on artificial intelligence and statistics, pp.
PMLR, 2020b.
Jiawang Bai, Baoyuan Wu, Yong Zhang. Yiming Li, Zhifeng Li, and Shu-Tao Xia. Targeted attack against deep neural networks flipping limited weight bits. arXiv preprint arXiv:2102.10496. 2021.
Jiawang Bai, Kuofeng Gao, Dihong Gong, Shu-Taa Thu Xia, Zhifeng Li, and Wei Liu. Hardly perceptible trojan attack against neural networks with bit flips. In ECCV, 2022.
Jiawang Bai, Baoyuan Wu, Zhifeng Li, and Shu-Ta Tao Xia. Versatile weight attack via flipping limited bits. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023a.
Jiawang Bai, Kuofeng Gao, Shaobo Min, Shu-Tao Xia, Zhifeng Li, and Wei Liu. Badelip: Trigger- aware prompt learning for backdoor attacks clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24239-2423 2024.
Yijie Bai, Yanjiao Chen, Hanlei Zhang, Wenyuan Xu, Haiqin Weng, and Dou Goodman. (VILLAIN): Backdoor attacks against vertical split learning. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 2743-2760, 2023b.
Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data with informed adversaries. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1138-1156. IEEE, 2022.
Moran Baruch, Baruch Gilad, and Yoav Goldberg. A Little Is Enough: Circumventing Defenses For Distributed Learning. In NeurIPS, 2019.
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In ICML, 2019.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 1ood-101-miningg discriminative compo- nents with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp. 446-461. Springer. 2014.
Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining, pp. 160-17 172. Springer, 2013.
X. Cao, J. Jia, Zhang, and Gong. Fedrecover: Recovering from poisoning attacks in federated learning using historical information. In 2023 2023 IEEE Symposium on Security and Privacy (SP) (SP), pp. 326-343, Los Alamitos, CA, USA, may 2023. IEEE Computer Society. doi: (0.11099/P462155222...0000 2022000099. htthttps:: /at.iiiivv... org/ httgg//p/p//////0/ /2t//2210..o 10936.0366.000
Xiaoyu Cao, Minghong Fang. Jia Liu, and Neil Zhenqiang Gong. FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping. In NDSS, 2021.
Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning, arXiv preprint (00t:2//06006666) 2021.
11

---

Nicholas Carlini, Chang Liu, Jernej Kos, Ülfar Erlingsson, and Dawn Song. The Secret Sharer: Measuring Unintended Neural Network Memorization Extracting Secrets. 1802 2018.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270, 2023.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv: 171..555,, 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Wenlong Deng, Christos Thrampoulidis, and Xiaoxiao Li. Unlocking the potential of prompt-tuning in bridging generalized and personalized federated learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6087-6097, 2024.
Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhengiang Gong. Local Model Poisoning Attacks to Ferzantine-Robus Federated Learning. In USENIX, 2020.
Li Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop. pp. 178-178. IEEE, 2004.
Chong Fu, Xuhong Zhang, Shouling Ji, Jimyin Chen, Jingzheng Wu, Shanqing Guo, Jun Zhou, Alex X Liu, and Ting Wang. Label inference attacks against vertical federated learning. In 31st USENIX security symposium (USENIX Security 22), pp. 1397-1414, 2022.
Kuofeng Gao, Jiawang Bai, Baoyuan Wu, Mengxi Ya, and Shu-Tao Tao Xia. Imperceptible and robust backdoor attack in 3d point cloud. IEEE Transactions on Information Forensics and Security, 19: 1267-1282, 2023.
Tianyu Gu, Brendan Dolan-Gavit, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv:/708.060 06733, 2017.
Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 1147230-47244 2019.
Tao Guo, Song Guo, and Junxiao Wang. Pfedprompt: Learning personalized prompt for vision- language models in federated learning. In Proceedings of the ACM Web Conference 2023, pp. 1364-1374, 2023a.
Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Promptil: Let federated participants cooperatively learn prompts instead of models-federate learning in age of foundation model. IEEE Transactions on Mobile Computing, 2023b.
Asif Hanif, Fahad Shamshad, Muhammad Awais, Muzammal Naseer, Fahad Shahbaz Khan, Karthik Nandakumar, Salman Khan, and Rao Muhammad Anwer. Baple: Backdoor attacks on medical foundational models using prompt learning. In International Conference Medical Image Intervention, pp. 443-453. Springer, 2024.
Computing and Computer-Assinn
Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 2043-2055 2059. IEEF, 2022.
Peter Kairouz, Brendan McMahan, Brendan Avent, et al. Advances and open problems in federated learning.
2019.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. arXiv preprint arXiv:2210.03/ 2022.
12

---

Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks pre-trained models. arXiv preprint arXiv:2004.066 066600, 2020.
Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In ICCV, 2021.
Siyuan Liang, Mingli Zhu, Aishan Liu, Baoyuan Wu, Xiaochun Cao, and Ee-Chien Chang. Badclip: Dnal-enbeddiig guided backdoor attack on multimodal contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24645 22654,, 2024.
Yang Liu, Yan Kang, Tianyuan Zou, Yanhong Pu, Yuanqin He, Xiaozhou Ye, Ye Ouyang, Ya-Qin Zhang, and Qiang Yang. Vertical federated learning: Concepts, advances, and challenges. IEEE Transactions on Knowledge and Data Engineering. 36(7):3615-3344 2024.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1306.5 2013.
H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. learning of deep networks from decentralized data. In AISTATS, 2017.
Communication
EI Mahdi El Mhamdi, Rachid Guerraoui, and Sébastien Rouault. The Hidden Vulnerability of Distributed Learning Byzantium. In ICML, 2018.
Thien Due Nguyen, Phillip Rieger, Huili Chen, Hossein Yalame, Helen Möllering, Hossein Fereidooni, Samuel Marchal, Markus Miettinen, Azalia Mirhoseini, Shaza Zeitouni, et al. (FLAME}: Taming backdoors in federated learning. In 31st USENIX security symposium (USENIX Security 22), pp- 2022.
Thuy Dung Nguyen, Tuan A Nguyen, Anh Tran, Khoa D Doan, and Kok-Seng Wong. Iba: Towards irreversible backdoor attacks in federated learning. Advances in Neural Information Processing Systems,
2023.
Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. NeurlPS, 2020.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference computer vision, graphics & image processing, pp. 722-729. IEEE, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and patlern recognition, pp. 3498-3505. IEEE, 2012.
pytorch. PyTorch Documentation. https://pytorch.. org./ 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp- 8748-8763 PMLR, 2021.
Esha Sarkar, Hadjer Benkraouda, Gopika Krishnan, Homer Gamil, and Michail Maniatakos. Facehack: Attacking facial recognition systems using malicious facial characteristics. IEEE Transactions on Biometrics, Behavior, and Identity Science, 4(3):361-37) 2021.
V. Shejwalkar, A. Houmansadr, Kairouz, and D. Ramage. Back to the drawing board: critical eval- uation of poisoning attacks on production federated learning. In 2022 2022 IEEE Symposium on Se- curity and Privacy (SP) (SP), pp. 1117-1134, Los Alamitos, CA. USA, may 2022. IEEE Computer Society. doi: 109/1/099444444..000 20wp:4462222.22220000000000 URL ftttps://dois :ett::://oo..eeeccpuueeeerooooo www.ooptteetree.cccc...... crg/10.1109/S$0 42244 2022. 00065
Virat Shejwalkar and Amir Houmansadr. Manipulating the Byzantine: Optimizing Model Poisoning Attacks and Defenses for Federated Learning. In NDSS, 2021.
Virat Shejwalkar, Lingjuan Lyu, and Amir Houmansadr. The perils of learning from unlabeled data: Backdoor attacks on sumi-supervieed learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4730-4740, 2023.
13

---

Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning? NeurIPS FL Workshop, 2019.
Chandra Thapa, Pathum Chamikara Mahawaga Arachchige, Seyit Camtepe, and Lichao Sun. Splitfed: When federated learning meets split learning. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 8485-8493, 2022.
Alexander Turnrr Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv:19/2.027711 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv: 1812.00564 2018.
Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. In NeurIPS, 2020.
webankcredit.
Utilization of FATE in Risk Management of Credit in Micro Enterprises. rirks-
Small and
https://www, fedai. org/ cases//ass////////
in- sma mnd-mmcco-enterrrri
2019.
Yau Weng, Minh Hoang, Lam Nguyen, My T Thai, Lily Weng, and Nghia Hoang. Probabilistic federated prompt-tuning with non-iid and imbalanced data. Advances in Neural Information Processing Systems, 37:81933-81958, 2024.
Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. DBA: Distributed backdoor attacks against federated learning. In ICLR, 2019a.
Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. arXiw:1802.10116, 2018
Generalized byzantine-tolerant sgd.
Cong Xie, Sanmi Koyejo, and Indranil Gupta. Fall of empires: Breaking Byzantine-tolerant SGD by inner product manipulation. arXiv: 1rXhv:1903.03990 03090, 2019b.
Xiong Xu, Kunzhe Huang, Yiming Li, Zhan Qin, and Kui Ren. Towards reliable and efficient backdoor trigger inversion via decoupling benign features. In ICLR, 2024.
Mengxi Ya, Yiming Li, Tao Dai, Bin Wang, Yong Jiang, and Shu-Tao Xia. Towards faithful xai evaluation via generalization limited backdoor watermark. ICLR, 2024.
Dong Yin, Yudong Chen, Kannan Ramehandran, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In ICML, 2018.
Yi Zeng, Won Park, Z Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 16473-16481, 2021.
Hangfan Zhang, Jinyuan Jia, Jinghui Chen, Lu Lin, and Dinghao Wu. A3f1: backdoor attacks to federated lcarning, Advances in neural information processing systems, 36: 61213-61233, 2023.
Adversarially adaptive
Zaixi Zhang, Xiaoyu Cao, Jinyuan Jía, and Neil Zhenqiang Gong. Fidetector: Defending federated learning against model poisoning attacks via detecting malicious clients. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2545-2555, 2022.
Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Communication- efficient and preseeiiin prompt tuning in federated learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
14

---

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 168 16816-1622,, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 130(9))2337-2348, 2022b.
15

---

Appendix
We provide additional information for our paper, SABRE-FL: Selective and Accurate Backdoor Rejection for Federated Prompt Learning, in the following order:
Terminology/Teehnqques (Appendix
Additional Implementation Details (Appendix
Experimental Setup (Appendix C
Additional Results (Appendix
Rebuttal (Appendix
A
TERMINOLOGY/TCCHNLLOGGE
A.1
CLIP: CONTRASTIVE LANGUAGE-IMAGE PRETRAINING
CLIP, short for Contrastive Language- Image Pretraining, is a type of multimodal machine learning model developed by OpenAl Al Radford al. (021). "Multimodal" means it can process and relate information from two different types of inputs, in this case, images and natural language. Models like CLIP are referred as Eision-language models (VLMs) because they jointly understand both visual and textual information. CLIP was trained on a large dataset of 400 million (image, text) pairs collected from the internet. The idea behind CLIP is simple but powerful: given an image and sentence, the model learns to tell whether the sentence correctly describes the image. For example, given a photo of cat and several captions like "a cat," " "a dog," " or "a painting." CLIP learns to match the correct caption to the image. This is done using technique called contrastive learning, where the model pulls together matching image-text pairs and pushes apart mismatched ones the embedding space.
CLIP has two components: An image encoder (c.g., a Vision Transformer or ResNet) that converts images into high-dimensional vectors. A text encoder (e.g.. a Transformer) that converts sentences into vectors in the same space. After training, CLIP can be used for zero-shot classification, where is given list of possible text labels and an image, and it predicts which label best matches the image. This makes CLIP very versatile for downstream tasks, i.e., tasks that are different from the model's pretraining objective, such as object classification, image retrieval, OCR, or even robotics. During testing, CLIP matches a given test image with the best matching class label (converted into a prompt like "a photo of a class"). In summary. CLIP a ceneral-purpose vision-language model that learns a shared representation space for images and text without needing explicit labels. It serves as the foundation for prompt learning, which allows users adapt CLIP to new tasks more effectively.
A.2 PROMPT LEARNING
A prompt is a piece of text that is used guide model's predictions. In language models (like GPT), a prompt might be a sentence like "Translate this to French: Hello," and in CLIP, it might be "a photo of dog." " In the original CLIP setup, hand-crafted prompts like "a photo of a class" are used during lesting to convert text labels into embeddings. However, manual prompts are often suboptimal as they rely on human intuition and may not generalize well across tasks datasets. This led the idea of prompt learning, where instead of using fixed textual prompts, we learn soft prompts, i.e., a set of trainable vectors that replace or augment the context prompt. These prompts are optimized during training to improve model performance on a given downstream task.
The pioneering work in this area is CoOp (Context Optimization) Zhou et al., (2022b) which intro- duced learnable prompts for vision-language models like CLIP. In CoOp, the prompt is represented as a series of learnable embeddings [0i, U2, UN). which are prepended to each class name (e.g..
" "[v1] [v2]
[vN] dog"") and passed through the text encoder. These prompts are optimized using a small amount of labeled data. Prompt learning has several advantages: (I)It avoids fine-tuning the entire backbone, making it computationally efficient. (2) It adapts the model to new tasks with only a few training examples (few-shot learning). (3) It retains the generalization power of the pretrained model while specializing it for a specific task. Some common prompt hyperparameters
16

---

include: (1)Context length (N): the number of learnable prompt vectors prepended to the class name. (2)Number of shots: how many labeled examples per class are used for training. (3)Class token position: whether the class label appears at the start, middle, end of the prompt. Increasing the number of shots typically improves accuracy because the model sees more training examples per class, allowing the prompt learner to better capture the features that distinguish different categories. However, prompt learning often performs well even in low-shot settings, making it ideal for domains with limited labeled data.
A.3 BADCLIP
BadCLIP is a backdoor attack framework proposed in a CVPR 2024 paper [aii et al. (2024), designed to evaluate the vulnerability of prompt-learing-basee vision-language models like CLIP. Unlike traditional backdoor attacks that rely on visible patterns or simple data poisoning, BadCLIP crafts visually imperceptible noise triggers that manipulate the internal behavior of the model during both training and inference. Similar to CLIP, BadCLIP predicts the correct label by comparing image features to text features derived from prompts (e.g.. "a photo of a dog"). In the presence of a backdoor, a small adversarial noise pattern (trigger) is added to the input image. This trigger is optimized during training to cause the image encoder shift the image embedding closer the text embedding of an attacker-specified target class (e.g.. "cat"), while remaining visually indistinguishable to humans. BadCLIP also adapts the prompt vectors in trigger-aware manner. That is, both image features and context vectors are conditioned on the presence of the backdoor trigger, making the backdoor more robust and more likely to survive training. During inference, even a clean image given a trigger, the poisoned model misclassifies it as the target class due to embedding-leww drift.
More formally, given a clean image and trigger t, the backdoored input x x Q results in an image embedding [(=") that is closer to the prompt-conditioned text embedding of the target class g([V, )+)) than to its true label g ((V. C.y) The model predicts the target class even though the visual appearance corresponds to y. BadCLIP is the first backdoor framework using noise-base based triggers specifically designed for prompt-tuned CLIP models. Its key insight is that backdoor signals not limited to the input space but can be embedded into CLIP's latent space, making them both stealthy and effective. SABRE-FL builds on this idea, extending it to the federated learning setting.
A.4 PRIVACY LEAKAGE
Recent work has demonstrated that it is possible to reconstruct input data from machine learning models Balle et al. (2022); Carlini al. (2023) 2018). These attacks are known as reconstruction attacks. However, such attacks typically require certain strong assumptions. For example, Balle et al.) (202)) consider very strong adversary that knows several data points as well as the weights of the model.
SABRE-FL operates entirely in the representation space of a frozen CLIP encoder, meaning the image encoder is never updated with client-specific data. As result, the embeddings remain generic and task-agnostic, optimized for cooss-modal alignment, not input reconstruction. This design choice significantly reduces the risk of privacy Icakage, CLIP embeddings not trained to retain high-frequency or spsecfc image details.
While representation-llvell inversion remains an evolving area of research, current attacks often as- sume more favorable conditions than those present in SABRE-FL. Nevertheless, we acknowledge the broader risk and consider our design reflect a privacy-utille tradeoff: by accepting lightweight rep- resentation sharing with a fixed encoder, we achieve robust backdoor detection without compromising raw inputs or task-specific outputs.
17

---

B ADDITIONAL IMPLEMENTATION DETAILS
B.1 DETECTOR THRESHOLDING AND CLIENT FILTERING
In the main paper, we define the detector score Sş for each client C₁ as the mean classification output over its submitted embeddings:
S
Σ D(
VLk
where (.)) is binary classifier that outputs for poisoned embeddings. While this naturally allows for threshold-bases filtering (i.e., flagging clients for which S₃ > T), in practice we adopt a more stable rank-base heuristic.
Specifically, in each communication round, we assume m out of n clients may be malicious, and we remove the clients with the highest number of flagged embeddings (or highest S₁ scores). This avoids the need to hand-tune a static threshold т and reflects standard assumption in robust FL defense literature, where is typically known bounded Yin al. (2018); Xie et al. (2018). This rank-based heuristic is consistent with our earlier detector formulation and preserves the intended semantic interpretation of S& as client-level anomaly score.
C EXPERIMENTAL SETUP
C.I MODEL AND ATTACK SETTINGS
We use the CLIP model in a similar style as that of Bai et. al Bai et al. (2024). ViT-B/16 is used as the image encoder. The pretrained weights are taken from CLIP's released models Radford et al. (202T). We use a context length N 4, total number of epochs as 10, where and cosine learning rate scheduler with initial learning rate of 0.002. Unless specified otherwise, we keep the number of shots to be trigger optimization for epochs, and an SGD optimizer. The maximum noise strength, e, for the backdoor trigger is chosen to be 4. Similar to BadClip, the first class of every dataset is chosen as the target class during the attack.
a warmup epoch,
C.2 DATASETS
We use datasets that are used in CoOp Zhou et al. (2022b) and BadCLIP Bai et al. (2024). We use the same dataset configuration files they provide. The datasets use in our experiments are:
Caltech-100 101 Fei-Fei et al. (2004) is standard object classification dataset consisting of 9,146 images across 101 object categories and a background class. It has the license CC BY 4.0. Each category contains between 40 and 800 images of objects taken from varying viewpoints and backgrounds. The dataset is known for its moderate intra-class variation and has been widely used in evaluating vision models, especially in low-shot and few-shot learning settings. our work, we use Caltech-101 101 an out-of-distribution (OOD) dataset to train our backdoor detector. Importantly, this dataset is disjoint from the ones used in federated training, allowing us to test whether our detector generalizes across domains.
Flowers- Nilsback & Zisserman 2008) is a fine-grained classification dataset consisting of 8, 189 images of flowers categorized into 102 species. Each class contains between 40 and 258 samples. The high inter-class similarity and fine-grained nature of the dataset make it challenging benchmark for vision-language models.
The Oxford-III Pets dataset Parkhi et al. (0012 contains 7,349 images of 37 breeds of cats and dogs. Each class includes approximately 200 images captured varied poses, lighting conditions, and backgrounds. The dataset presents a mix of inter-class similarity and intra-class diversity, making suitable for testing the robustness of prompt learners in federated setups. It is available under the license CC BY-SA 4.0.
DTD Cimpoi et al. (2014) (Describable Textures Dataset) is cexture-centric classification dataset with 5,640 images labeled across 47 human descrrbbbee texture attributes such as "bumpy." "scaly." " "striped." The dataset emphasizes mid-level visual cues and is used in evaluation test.
18

---

whether prompt- based FL models maintain robustness when the notion of class is not strictly object-centric.
FGVC Aireraft Maji et al. (2013) contains 10,000 images of 100 aircraft variants grouped by manufacturer and model. It is fine-grained classification dataset that introduces significant challenges due to subtle inter-class differences and high intra-class consistency. We include it to assess whether backdoor attacks are effective even in domains where prompt learners must capture nuanced visual differences.
Food-101 Bossard et al. (2014) consists of 101,000 images across 101 food categories. The dataset exhibits significant visual diversity, both within and across classes, and is commonly used to benchmark image classification performance under real-world visual noise and clutter. It serves as one the more large-scale and diverse benchmarks our federated evaluation.
C.3 DEFENSE METHODS
We compare our technique with four popular defense techniques. Trimmed mean Yin et al. (2018): Xie et al. (2018) is widely used defense in FL, where the server receives updates from each client, sorts them across each dimension, and then discards the m smallest and lowest values across each dimension. Here, m is the number of malicious clients. Median Yin et al. (2018) is another popular defense mechanism, where the global model is computed by taking the dimension-wing - median of the client updates. form-bounding (unn al. (2019) clips the values of client updates to a certain value so they do not exceed that threshold. This threshold is computed by taking the median value of the client updates. FLAME Nguyen et (2022) is more complex defense that first clusters the clients into benign and malicious groups using hdbscan Campello et al. (2013), clips them at certain threshold, and adds noise to the model parameters to make them resilient backdoors.
C.4 DETECTOR TRAINING
We train a detector D Rd + (0, 1} to minimize binary cross-entropy loss over this embedding dataset. The model architecture is two-layer multilayer perceptron (MLP) with a hidden layer of size 128 and ReLU activation. takes as input CLIP image embeddings 2; E Rd (with d = 512) and outputs logits corresponding the clean or backdoored class. Optimization is performed using the Adam optimizer with a learning rate of 1 x 10-3 for 20 epochs, and batch size 64. The detector's 's file size is few MBs.
To evaluate cross- dross-domain generalization, we test the trained detector on separate held-out datasets, namely Oxford Flowers, Pets. DTD, FGVC Aireraft, and Food101, each containing mix of clean and poisoned embeddings. Despite being trained on a single auxiliary dataset, the detector consistently achieves 90% accuracy these unseen domains. This supports hypothesis that poisoned embeddings exhibit consistent statistical signature in CLIP space, independent of the underlying dataset class distribution.
C.5 RESOURCES
We used Py pytorch for our coding on Linux-based system. For running experiments, we use our university cluster that has different types of GPUs. Most of our experiments were performed on 12 GB NVIDIA TITANX GPUs. The run time of the experiments depended upon the dataset used, number of shots, and number of clients.
D ADDITIONAL RESULTS
D. T-SNE
We show the t-SNE plot of Oxford Flowers clean and backdoored embeddings in Figure
D.2 VARYING NUMBER OF SHOTS
We show the impact of varying the number of shots on all five datasets in Figure
19

---

Clean
Backdaored
20
-20
-40
60
40
20
20
40
60
Figure 7: t-SNE Flowers
Cas for Iafrree
Pedicio for Enhereed
(a) Flowers
(b) Pets
(c) DTD
fackder Acourgy pods Koet
Oan't Beciderr Acungg Faedica
100
1-10
Minderr Io. Dihees
(d) FGVC Aircraft
(e) Food101
Figure 8: Our defense consistently reduces backdoor success without degrading clean performance, the number of shots increases.
E REBUTTAL
In this section, we present new experimental results conducted in response to reviewer feedback. These include visualizations of learned triggers (Figure 9), evaluations under varying data heterogene- ity using Dirichlet sampling (Table 4), ablations on trigger strength and optimization steps (Tables 5-6), robustness analysis under imperfect excessive client filtering (Tables 7-8), and detector generalization across auxiliary datasets (Table 9). Collectively, these results further strengthen our claims regarding the effectiveness, generalizability, and practical robustness of SABRE-FL.
20

---

(a) Backdoored (Caltech)
(b) Trigger (Caltech)
(c) Backdoored (Pets)
Trigger (Pets)
(e) Backdoored (DTD)
(I) Trigger (DTD)
Backdoored (Aircraft)
(h) Trigger (Aircraft)
(i) Backdoored (Food101)
(j) Trigger (Food101)
Figure 9: Visual examples of backdoored inputs and corresponding learned triggers. Each row shows a dataset-specific backdoored image (left) and the additive noise trigger alone (right).
Table 4: Clean and backdoor accuracy across varying Dirichlet values (heterogeneity levels), with and without SABRE-FL. Lower indicates higher non-IID-ness.
Calueh
Pris
DTD
Aireaii
Foodini
Defense
SABRE
No
SABRE
Defense
SABRE
SABRE
8.9
V1.7
22.3
97.7
8.4
95.2
58
95.3
4.
618
9.3
64.5
7.1
319
66.7
12.5
0.1
91011
36.0
90.4
3,8
0.7
07.3
32.1
07.2
8.5
94.5
10.8
94.4
499
61.8
11.3
64.1
9.6
30.7
55.0
30.9
33.0
90.0
14.7
90.3
2.4
0.5
07.2
104
97.7
8.3
94.4
17.9
95.3
60.
25.3
650
1.6
107
85.
89%
89.5
33.2
0.3
20.5
19.9
21

---

Table Effect of trigger strength (e scaling) and SABRE-FL. defense on clean accuracy (CA) and backdoor accuracy (BA). Best BA (lower is better) is in bold.
Caltech
||
Aircraft
Food101
Setting
BA
CA
||
BA
|
CA
2 (Na Defense)
97.2
51.4
93.9
6.0
64.9
51.4
31.2
92.7
90.1
33.5
0.5 (No Defense)
97.1
8.7
92.1
27.3
64.6
27.8
30.5
80.0
89.8
4.5
SABRE.
97.1
7.6
4.6
64.4
4.6
32.0
17.2
90.7
3.9
0.5 e SABRE-FL
97.1
94.6
4.3
64.4
5.8
32.0
1.0
90.7
Table 6: Effect of trigger optimization steps (epochs) on clean accuracy (CA) and backdoor accuracy (BA). Higher CA and lower BA are better.
Setting
||
Caltech
Pets
DTD
Aireraft
Food101
||
CA
BA
CA
BA
CA
BA
CA
BA
CA
BA
no defense (1 epach)
96.9
31.6
94.1
7.0
64.7
30.3
28.9
61.7
90.2
3.0
defense epachs)
SABRE.FI.C epoch) SABRE: FL (S epachs)
94.4
31.6
97.3
8.1
94.7
4.3
66.2
2.7
31
0
90.5
3.1
Table 7: Effect of imperfect client filtering: we intentionally do not remove some malicious clients and evaluate SABRE-FL's robustness on Pets and DTD. Clean accuracy (CA) drops mildly, while backdoor accuracy (BA) rises with more undetected attackers.
||
Malicious Clients Not Removed
||
CA
BA
CA
92.1
4.7
62.2
11.3
91.1
7.2
61.0
14.1
90.5
9.1
61.1
19.7
4
89.8
10.1
60.5
25.5
Table 8: Effect of over-pruning: SABRE-FL removes the correct number of malicious clients but also accidentally filters out 1-2 benign clients. Despite this, clean accuracy (CA) remains stable and backdoor accuracy (BA) stays low.
Setting
Caltech
Pets
DTD
Aircraft
Food101
|| CA
BA
CA
BA
CA
BA
CA
BA
CA
BA
2 Malicious Benign Removed
97.3
6.6
94.8
4.5
64.5
6.8
33.4
0.4
90.5
2.0
2 Malicious Benign Removed
97.4
7.3
94.9
63.5
5.4
31.1
17.2
90.3
2.1
Table 9: Evaluating SABRE-FL when the detector trained Flowers instead of Caltech. Despite the shift in auxiliary dataset, the defense maintains strong generalization across tasks, reducing backdoor accuracy (BA) while preserving clean accuracy (CA).
Setting
Caltech
Pets
DTD
Aircraft
Food101
CA
BA
CA
BA
CA
BA
CA
BA
CA
BA
No Defense
97.2
58.2
92.1
12.1
67.6
27.3
32.5
91.5
90.0
20.6
(Trained Flowers)
22


Reviews:
[{"review_id": "9R78BQHQw1", "reviewer": "Reviewer_D5Ed", "summary": "This paper studies backdoor attacks in Federated Prompt Learning (FPL) and introduces SABRE-FL, a defense framework that detects and filters poisoned client updates using representation-space anomaly detection. The key idea is to train a binary detector on CLIP embeddings from an auxiliary dataset which includes clean and triggered/poisoned samples, so the model can identify abnormal embedding patterns corresponding to malicious clients' updates during FL aggregation. \n\nThe method is evaluated on five datasets, which are Flowers, Pets, DTD, FGVC Aircraft, and Food101 under varying malicious client ratios, showing that SABRE-FL achieves low backdoor success rates while preserving clean accuracy.", "questions": "Please revise the methodology section based on the weaknesses mentioned above. Also, adding some more important experiments should strengthen the contribution of this paper.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper studies backdoor attacks in federated prompt learning (FPL), where only prompt parameters, not full model weights, are shared; this is timely and relevant as CLIP-style adaptations in FL are increasingly used.\n2. The method shows potential generalizability: a detector trained on Caltech-101 transfers to datasets not seen during training.\n3. The paper provides clear motivation and visualization that support the defense’s intuition; the results look promising in tackling the backdoor attack tested.", "weaknesses": "1. The methodology section lacks important details.\n(i) The paper describes the trigger as a learnable noise pattern but does not explain how it is optimized, what loss function or parameters are used, and how it interacts with the local prompt updates (e.g., whether it uses SGD, PGD, or another generator).\n(ii) The defense critically depends on the parameter $m$—the number of clients excluded from aggregation each round—but there is no principled method or empirical guideline for setting this value.\n(iii) The detector is trained on an auxiliary dataset (Caltech-101) with synthetically poisoned samples, yet the paper provides little justification that this dataset captures the diversity of real backdoor triggers or resembles the adversary’s trigger design. The assumption that embedding deviations generalize across datasets and trigger types is unverified.\n\n2. Several important ablation studies are missing, such as varying $m$, testing under different non-IID settings, changing the auxiliary dataset, and exploring the effects of trigger strength, magnitude, and optimization steps.\n\n3. The paper does not visualize or analyze the learned trigger patterns that drive the backdoor. Showing how the trigger alters image embeddings or prediction confidence would make the mechanism more transparent.\n\n4. The evaluation focuses on a single type of learnable additive-noise trigger inspired by BadCLIP. It does not test against adaptive or structurally different triggers (e.g., patch-based, frequency-domain, sample-specific, or model-poisoning attacks). As a result, the claimed generality of SABRE-FL across attack mechanisms is not sufficiently demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HqCMym9s9B", "reviewer": "Reviewer_jAGJ", "summary": "The paper investigates the security vulnerabilities of Federated Prompt Learning (FPL) under backdoor attacks. The authors show that FPL models can be compromised when malicious clients inject visually imperceptible, learnable noise triggers into images, leading to targeted misclassification while maintaining high clean accuracy. To counter this, they propose SABRE-FL, a modular and lightweight server-side defense that employs an embedding-space anomaly detector—trained offline on out-of-distribution (OOD) data—to identify and filter poisoned prompt updates without accessing raw client data or labels. Experiments demonstrate that SABRE-FL significantly reduces backdoor accuracy while preserving clean performance across various datasets and FPL settings.", "questions": "Please address the aforementioned weaknesses, particularly by including qualitative visualizations, sensitivity analyses for m, comparisons to stronger defenses, and discussions of detector robustness under Non-IID and adaptive attack scenarios.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Pioneers the study of backdoor threats in the emerging FPL paradigm.\n- Introduces a well-motivated and FPL-specific backdoor mechanism based on learnable, imperceptible noise triggers.\n- Clear writing and strong organization, aided by effective visual explanations of both attack and defense designs.", "weaknesses": "1. The paper claims the noise triggers are *visually imperceptible*, but lacks direct image comparisons. Including visual examples (original vs. triggered) or a qualitative study would strengthen this claim.\n2. SABRE-FL removes the top-*m* suspicious clients, assuming *m* is known. An analysis of sensitivity to inaccurate estimates of *m* would clarify robustness in real-world settings.\n3. The distinction between the proposed attack and a federated adaptation of BadCLIP should be elaborated—what novel properties or mechanisms are introduced?\n4. The effect of data heterogeneity (Non-IID settings) on SABRE-FL’s detection performance is not well-studied; benign diversity may confound the embedding-based detector.\n5. SABRE-FL’s effectiveness depends on the defender’s ability to model known trigger behaviors when training its detector offline. However, this reliance makes it vulnerable to novel or unconventional backdoor types—such as semantic, geometric, or other model-poisoning attacks—that fall outside the learned embedding distribution.\n6. Additional comparisons to recent strong baselines such as **Deepsight** [1] or **BackdoorIndicator** [2] would better contextualize SABRE-FL’s improvements.\n\n[1]. Rieger, Phillip, et al. \"Deepsight: Mitigating backdoor attacks in federated learning through deep model inspection.\" arXiv preprint arXiv:2201.00763 (2022).\n\n[2]. Li, Songze, and Yanbo Dai. \"{BackdoorIndicator}: Leveraging {OOD} Data for Proactive Backdoor Detection in Federated Learning.\" 33rd USENIX Security Symposium (USENIX Security 24). 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f0V6qiIBwM", "reviewer": "Reviewer_4QUt", "summary": "his paper proposes a method named SABRE-FL to defend Federated Prompt Learning (FPL) against backdoor attacks. It takes the form of a server-side defense mechanism that uses a lightweight detector to filter malicious client updates. To achieve this, they employ an anomaly detector that operates in the CLIP embedding space, which is trained offline on out-of-distribution (OOD) data. The paper outlines the vulnerability of FPL to malicious clients injecting learnable noise triggers and proposes a solution to mitigate these issues. Their main contribution is SABRE-FL, a lightweight and modular defense framework, which leverages the consistent deviation produced by poisoned samples in the embedding space to identify and filter poisoned prompt updates. They conducted an empirical study of the attack across five diverse datasets, concluding that FPL is highly vulnerable to backdoor attacks while still maintaining high clean data accuracy. Finally, an empirical study of the defense was conducted, concluding that SABRE-FL demonstrates superior performance compared to four other baseline defenses, as it can reduce backdoor accuracy while preserving clean accuracy.", "questions": "1.If m is unknown or mis-specified (e.g., if the server underestimates the number of attackers), how does the defense performance of SABRE-FL (both CA and BA) degrade?\n2.I would like to see some discussion regarding the communication overhead introduced by SABRE-FL.\n3.Could the authors provide more insight into the results on the FGVC Aircraft dataset (Figure 8d)? In the 'no defense' setting, this dataset exhibits two extremes: the lowest Clean Accuracy (CA) and the highest Backdoor Accuracy (BA), and these results differ significantly from those of the other datasets.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This work presents the first systematic study of backdoor attack vulnerabilities within the Federated Prompt Learning (FPL) paradigm. This exploratory contribution is significant as it illuminates a critical and previously unexamined attack dimension.\n2.The paper proposes SABRE-FL, a novel server-side defense mechanism. The core of this mechanism involves using a lightweight MLP, trained offline on an out-of-distribution (OOD) dataset, to detect embedding-space anomalies.\n3.A key advantage of this defense is its generalizability. The experiments demonstrate that the detector, trained on a single auxiliary dataset, can effectively generalize and be applied across five other distinct task datasets.", "weaknesses": "1.SABRE-FL is essentially an anomaly detector. In heterogeneous (Non-IID) FL scenarios, natural shifts in data distribution are an inherent characteristic. The paper provides no evidence that the detector D can distinguish between malicious offsets caused by the attack and benign shifts arising from this data heterogeneity. This casts serious doubt on the method's effectiveness in realistic FL settings.\n2.The defense mechanism relies on an assumption that is difficult to satisfy in practice: the server must know the exact number of malicious clients m,a prior in each round to filter out the clients with the top-m highest scores. This assumption is unrealistic for most real-world scenarios.\n3.SABRE-FL (according to Algorithm 1) requires clients to upload the embeddings for all their local data in each round. This is likely to incur substantial communication overhead.\n4.The paper only provides a small-scale experiment (32 clients) in Appendix E.3 and lacks an evaluation on larger-scale federated networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0CLDWq4Ruo", "reviewer": "Reviewer_r8mg", "summary": "This paper studies the security of Federated Prompt Learning (FPL) and presents SABRE-FL, a defense framework that detects backdoored client updates using embedding-space anomaly detection. The authors first demonstrate that prompt-based federated systems, despite having a smaller attack surface than full-model FL, are highly vulnerable to imperceptible noise-trigger attacks. SABRE-FL then leverages CLIP embedding deviations to filter malicious updates without accessing raw client data or labels.", "questions": "1. Could authors provide the effectiveness of the method against adaptive attackers?\n2. Could authors justify the practicality of the pretraining phase?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel and timely problem. The problem of backdoor attacks in federated prompt learning is novel and interesting. The paper fills an important gap in understanding the security risks of adapting foundation models via prompt learning in a decentralized setting.\n2. Clear intuition and methodology. The core idea is intuitive and well-motivated. The defense operates in embedding space, aligning with the privacy constraints of FL.\n3. Strong empirical evaluation. Extensive experiments across five datasets and four defense baselines. SABRE-FL achieves the lowest backdoor accuracy while maintaining high clean accuracy.", "weaknesses": "1. Limited threat model. Only data poisoning is considered. Model poisoning or adaptive strategies are not analyzed. It is unclear whether an attacker aware of SABRE-FL could evade embedding-space detection. The assumption that the attacker controls 25% of all clients is generally considered to be high compared with existing literatures.\n2. Detector training assumptions. The detector is trained using poisoned embeddings generated on an auxiliary dataset. It is not obvious how a real-world server would obtain such poisoned examples to train the detector. The paper should justify the practicality of this pre-training phase.\n3. More explanation on comparison to FLAME and other baselines. FLAME also uses embedding-based filtering. The paper needs a clearer conceptual distinction and justification of why SABRE-FL is fundamentally stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)