MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Authors: Shixi Qin
Zhiyong Yang
Shilong Bao
Shi Wang
Qianqian Xu
Qingming Huang
Venue: ICML 2025 poster (2025)
Keywords: Backdoor Attack
Schrödinger Bridge
Primary area: social_aspects->security

Abstract: This paper focuses on implanting multiple heterogeneous backdoor triggers in bridge-based diffusion models designed for complex and arbitrary input distributions. Existing backdoor formulations mainly address single-attack scenarios and are limited to Gaussian noise input models. To fill this gap, we propose MixBridge, a novel diffusion Schrödinger bridge (DSB) framework to cater to arbitrary input distributions (taking I2I tasks as special cases). Beyond this trait, we demonstrate that backdoor triggers can be injected into MixBridge by directly training with poisoned image pairs. This eliminates the need for the cumbersome modifications to stochastic differential equations required in previous studies, providing a flexible tool to study backdoor behavior for bridge models. However, a key question arises: can a single DSB model train multiple backdoor triggers?  Unfortunately, our theory shows that when attempting this, the model ends up following the geometric mean of benign and backdoored distributions, leading to performance conflict across backdoor tasks. To overcome this, we propose a Divide-and-Merge strategy to mix different bridges, where models are independently pre-trained for each specific objective (Divide) and then integrated into a unified model (Merge). In addition, a Weight Reallocation Scheme (WRS) is also designed to enhance the stealthiness of MixBridge. Empirical studies across diverse generation tasks speak to the efficacy of MixBridge. The code is available at: https://github.com/qsx830/MixBridge.

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Shixi Qin
Zhiyong Yang'
Shilong Bao
Shi Wang?
Qianqian Xu
Qingming Huang 132
Abstract
the efficacy of MixBridge. The code is avail- able at: https://github: com/ qsx83// MixBridge.
This paper focuses on implanting multiple hetero- geneous backdoor triggers in bridge-bane based diffu- sion models designed for complex and arbitrary input distributions. Existing backdoor formula- tions mainly address single-attack scenarios and are limited Gaussian noise input models. To fill this gap, we propose Mi xBri igge, novel diffusion Schrödinger bridge (DSB) framework to cater to arbitrary input distributions (taking 12I tasks as special cases). Beyond this trait, we demonstrate that backdoor triggers can injected into MixBridge by directly training with poi- soned image pairs. This eliminates the need for the cumbersome modifications to stochastic dif- ferential equations required in previous studies, providing a flexible tool to study backdoor be- havior for bridge models. However, a key ques- tion arises: can single DSB model train mul- tiple backdoor triggers? Unfortunately, our the- ory shows that when attempting this, the model ends up following the geometric mean of benign and backdoored distributions, leading to perfor- mance conflict across backdoor tasks. To over- come this, we propose a Divide-and-Merge strat- egy to mix different bridges, where models are independently pre-trained for each specific ob- jective (Di di) and then integrated into uni- fied model (Merge). In addition, Weight Re- allocation Scheme (WRS) is also designed to en- hance the stcalthiness of MixBria i dge. Empirical studies across diverse generation tasks speak to
Warning: This paper contains model outputs that may offensive in nature.
1. Introduction
Diffusion models have demonstrated remarkable perfor- mance across various domains, including image (Ho et al., 2020; Dhariwal & Nichol, 2021; Ho & Salimans), text (Lou et al.). audio (Kong et al.), speech generation (Huang et al., 2022), and others (Wen et al., 2022). Alongside the success of diffusion models, potential risks have emerged, such as vulnerability to backdoor attacks. In a backdoor attack sce- nario, the attacker secretly implants triggers into victim model. Once unsuspecting users deploy the victim model in their applications, these triggers can be activated to produce harmful content (e.g., Not-Suitable-FF Work images).
So far, various backdoor attack methods have been devel- oped reveal the mechanism of backdoor attacks in diffu- sion models. Existing research primarily focuses on two spe- cific formulations: unconditional diffusion models (Chou et al., 2023; Chen et al., 2023a; Chou et al., 2024) and con- ditional Imxte (T2I) diffusion models (7hai et al., 2023; Struppek et al., 2023; Shan et al., 2024; Wang et al., 2025). However, we argue that two crucial scenarios remain unexplored in the current literature. First, existing backdoor formulations designed for diffusion-based models that exclusively take Gaussian noise input. Unfortunately, real-world tasks often require models to handle more ar- bitrary and complex input distributions, rather than pure noise. For example, tasks like super-resolution and image inpainting use images as input in an Image-to-Image (121) generation context. Second, most studies merely focus on the single backdoor attack. Yet in practice, malicious attack- ers may consider heterogeneous backdoor attacks, such as injecting multiple backdoors with different triggers simulta- neously, to enhance stealthiness and maximize the impact of their attack.
'School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 101408, China KKyy Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy Sciences, Beijing 100190, China 'Key Laboratory of Big Data Mining and Knowl- edge Management (BDKM), University of Chinese Academy of Sciences, Beijing 101408, China. Correspondence Zhiy- ong Yang <yangzhiyong21 uecaas.cccnn>. Qingming Huang <qmhuang @ccas.ac.n>>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To address these issues, our goal in this paper two-fold:

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
1) Investigate 121 backdoor attacks on diffusion mod- els across arbitrary input and output image distribu- lions. 2) Explore heterogeneous backdoor attacks in this context.
nign outputs when given clean input images (i.e., utility) and generating heterogeneous malicious outputs when input images contain triggers (i.e., specificity).
Our contributions can be summarized in three folds:
For 1), our proposed method is top of the bridge-based diffusion models (De Bortoli et al., 2021; Zhou et al.: Gushchin et al., 2024; Yang et al., 2021), which have garnered significant attention due to their flexibility in achieving transformations between arbitrary distributions. Specifically, we propose Mi MiBriige, an MoE-based 12I Schrödinger bridge (12SB) model (De Bortoli et al., 2021; Liu et al., 2023; Shi et al., 2024). Different from traditional diffusion models, our proposed method can now handle arbi- trary input image distributions. This completely avoids the need to implant triggers on the noises. Moreover, we theo- retically show that one only needs to train the Mi xBridge model directly with backdoor images (Prop. 4.1), where the stochastic differentiable equations (SDEs) could be im- plicitly learned. This differs significantly from previous studies (Chou et al., 2023; Chen et al., 2023a; Chou et al., 2024: Bao et al., 2025), which all require the cumbersome design of SDEs to implement diffusion backdoor attacks.
We propose MixBridge, Schrödinger Bridge (DSB) model, to study the backdoor attacks. To our knowledge, we are the first to study the backdoor attacks on the arbitrary diffusion bridge models.
121 diffusion
We propose a divide-and-men -erge training strategy to reconcile the performance conflict between benign and different malicious generation tasks.
We validate the performance of the proposed frame- work on ImageNet and CelebA, demonstrating that our model can generate high-quality benign and heteroge- neous malicious outputs.
2. Related Work
Diffusion Model. The diffusion model is widely used in im- age generation (Song et al., a;b; Meng et al., 2024). While early diffusion models (Ho et al., 2020) were designed for unconditional image generation, subsequent models incor- porated additional guidance enhance control over the generated content (Dhariwal & Nichol, 2021; Chao et al., 2021). Based on these, researchers propose Image-to-Imen Image (121) diffusion models, which take input images and generate guided outputs (Nichol et al., 2022; Sasaki et al., 2021). (Sa- haria et al., 2022a) applics conditional diffusion models to edit images, and (Saharia al., 2022b) uses a stochastic it- erative denoising process to achieve super-resolution. Apart from these, some studies have explored diffusion bridges for image generation (Shi et al., 2024; Zhou et al.). For instance, the diffusion Schrödinger Bridge (DSB) achieves an optimal transport between two distributions, suitable for I2I generation. In (Ye et al., 2024). DSB is applicd to solve the unpaired I2I translation between distinct image distributions. (Wang al., 2024b) employs DSB to perform super-resolution for medical images.
For 2), we find that it challenging to execute heteroge- neous backdoor attacks within single 12SB model. Specif- ically, if one uses a single model to fit the sample path of the benign distribution and all different types of backdoored distribution, then we prove that (Thm. 4.2) the algorithm follows the geometric mean of these distributions. Because the sample paths for benign and different backdoor tasks differ greatly in distribution, the geometric mean and most target distributions often lies from afar, degrading genera- tion quality for both benign and malicious generation tasks. To resolve this challenge, we propose divide-and-ment strategy to reconcile the performance conflict between dif- ferent tasks. In the divide stage, we use a task- specific warm-up strategy to pre-train different tasks separately. In the merge stage, we integrate the pre-trained model with the Mixture of Experts (MoE) framework. Another key is- sue is that a naive MoE framework tends assign much higher weight to the original pre-trained expert for the given tasks. By looking at the weight assignment statistics, the user can easily detect which experts are responsible for the backdoor attacks. To further improve the stealthiness of MixBridge, we propose a Weight Reallocation Scheme (WRS) to encourage uniform weight assignment, making the backdoor experts less noticeable to the victim user. In this sense, MixBri igge can retain the advantages of task- specific models while keeping the weight assignments in- conspicuous.
Backdoor Attacks for Diffusion Models. Early stud- ies on backdoor attacks mainly focus on the classification tasks (Gu et al., 2017; Doan et al., 2021), where a back- doored model produces predefined predictions only when the input contains the trigger. Recently, researchers have investigated backdoor attacks on different generative mod- els (Rawat et al., 2022; Jiang al., 2024). In this paper, we restrict our discussion to the cnffusiionbasele models, which aim to modify diffusion models so that the attacker can ma- nipulate outputs by activating hidden backdoors. Regarding the diffusion model, previous studies can be roughly divided into Text-to- Image (T2I) and unconditional diffusion attacks.
Finally, we validate the effectiveness of Mi filxBridge idge on the ImageNet and CelebA datasets. Our results demonstrate the model's dual capabilities: producing high-quality be-
2

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
The T2I diffusion model involves additional text as input, so the attacker can easily inject triggers into the text (Zhai et al., 2023; Struppek et al., 2023; Wang et al., 2025; Pan et al.,
parameterized by 0 can be trained as follows:
(Icter. t:
3z 20
|22
2023; Huang et al., 2023). Previous backdoor attack meth- ods on unconditional diffusion models (Chou et al., 2023; 2024; Chen et al., 2023a) generate benign images from a standard Gaussian noise and inject triggers into the Gaus- sian noise to induce the model generate backdoor outputs. In response backdoor attacks, some defense algorithms have been proposed (An et al., 2024; Guan et al., 2024; Mo et al., 2024; Yang et al., 2023b; 2022), which detect backdoor attacks by analyzing the input noise distributions.
[()) = 12-2((011) (.m.
Tr
(I)
where 2, q(x; 20 x1) can be considered as a linear com- bination of 2o and æ1. After training, the target examples can be sampled in the same way the standard diffusion model according to DDPM (Ho et al.. 2020). This way, the target image 2o can be recursively deduced from the input image œ1. Details of the DSB problem and 12SB are discussed in Appendix A.
A Summary of Ours. The relationship between our pro- posed backdoor attack and prior studies can be concluded as follows. In terms of the similarities, our backdoor attack method is built upon the generative diffusion framework, which solves 12I tasks in the benign case, and generates malicious outputs if the input contains a trigger. However, it differs in four key aspects. First, unlike early backdoor attacks that induce misclassifications, the MixBridge aims to generate target backdoor images. Second, the MixBridge targets the bridge-based diffusion models that directly take images as inputs, while previous studies mainly focus on unconditional or T2I diffusion models that start from a stan- dard Gaussian noise. Third, we consider heterogeneous backdoor attacks against diffusion models, while previous studies consider a single backdoor attack only. Fourth, ex- isting defenses for diffusion models rely on the assumption that the diffusion process begins with Gaussian noise. As result, they may not effectively mitigate our proposed 121 backdoor attack.
3.2. Heterogeneous Backdoor Attacks
The backdoor attacker's 's goal is to manipulate a diffusion model's output to produce malicious results. Previous stud- ies have explored single-pattern backdoor attacks in diffu- sion models and some associated defense methods. How- ever, in real scenarios, a model is likely to face attacks in various patterns when an aggressive attacker injects mul- tiple, distinct triggers into the model, which we call het- erogeneous attacks. Seeing this new threat, we explore heterogeneous backdoor attacks on top of diffusion bridge models to deepen our understanding of the inner mechanism. To begin, we outline the attacker's setup as follows.
A Attacker's Capacity and Goal:
We assume that the at- tacker has full control over the training dataset and the training process. Once the model is trained, the attacker then uploads the poisoned model to public marketplace. When people deploy this poisoned model, the attacker aims to activate the embedded backdoors to generate malicious images predetermined by the attacker while keeping others unaware. These malicious outputs may cause ethical and legal issues for the user hosting the model.
3. Preliminary
In this paper, built upon the bridge-based diffusion model, we aim to investigate the 12I backdoor attacks as a rep- resentative case of the general diffusion process between arbitrary distributions. In this section, will first introduce the 121 Schrödinger bridge we implement and then outline the setups for the backdoor attacks.
To achieve the goal, the model must satisfy two critical requirements as discussed in previous studies: utility and specificity (Chou et al., 2023; 2024). The utility requires the poisoned model to behave normally when processing clean inputs, generating high-quality benign outputs. This ensures the poisoned model does not arouse suspicion. On the other hand, when the input images contain triggers, the specificity enables the model to produce malicious outputs, such as Not-Suitable-Fot Forroo-k (NSFW) images (Zhang et al., 2025), copyright infringement images (Wang et al., 2024a), etc.
3.1. Image-to-Image Schrödinger bridge
Our work is primarily built on a recently emerging ef- feetive diffusion bridge model called the Image-to- toage-to-iimgee Schrödinger bridge (I2SB) (Liu et al., 2023). The goal of 12SB is to learn the nonlinear diffussinn bridge between two given distributions, which can be used in various down- stream image restoration tasks, such as super-resolution and image inpainting. To be specific, let # PB repre- sent the input image of 12SB and Zy PA represent the corresponding target image'. The 12SB model < 0)
4. MixBridge
In this section, we first introduce the backdoor attack prob- lem for diffusion models. Then, we study how to inject backdoor triggers into the typical diffusion bridge model (I2SB). Taking a step further, we explore how to conduct het-
'The generation process is reversed from to 0.
3

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Backdoor Expert
Input
Single Musiel
MixBridge
resotution
Fake Face (Backdoor)
reso
Super-resolution (Benign)
Benign Expert
Taked
NSFW (Backdoor)
Backdoor Expert
NSeW
Generation Process
Figure 1: Overview of the generation process of the diffusion model. (Left) The model processes images from an imput distribution and generates output images along distinct diffusion trajectories. Notably, both the input and output distributions are mixture distributions. While the input images maintain a degree of similarity, there exists significant disparity among the heterogeneous output distributions. (Right) With various 5, injected into the input image, the model generates different outputs. Obviously, the quality of the output image generated by the Mi xBri igg is much better than the single model.
erogeneous backdoor attacks in this context. Due to space constraints, all proofs are deferred to Appendix B.
resolution task are naturally the high-resolution images and the corresponding low-resolution images.
However, there is innate relationship between clean and malicious images. For each backdoor attack, we prepare
4.1. Problem Formulation
Nn. To form appropriate malicious dataset Dm {m² jj=1
In a typical diffusion process over E [0, 1]. the model pro- cesses clean input image ø generate benign output image æg.. To manipulate the model, an attacker may inject backdoor trigger ō into æt,, creating poisoned input ar"- This input then guides the model produce a malicious out- put image z as specified by the attacker. Mathematically, this poisoning process is formalized as follows:
pairs for attacks, we select the malicious image xB; from Dm for each clean input based on the proximity evaluated by Euclidean distance: o = arg min kE
|| k||2-
Given the settings above, we argue that a critical challenge in this context managing the mixed data distribution of clean and poisoned images. The overall data distribution time step t is modeled as:
o = (1- M) of + M 6,
(2)
where M denotes the binary mask corresponding to the trigger, and zf is the resulting poisoned input. Typically, backdoor triggers are subtle and visually inconspicuous, such as small black patch in the corner of the input image, as illustrated in Fig. (left).
M =
p (xx)) p(c) p (xffe)
p(i) P 21)
(3)
where p(æ:)) represents the probability density function of the mixture distribution time t. Here, 2f and 2f de- note the clean and poisoned images under the i-th attack, respectively. The terms p(c) and p(i) are proportion for the clean and i-th poisoned images, with the constraint p(c) + ΣM p(i) = 1 ensuring a valid distribution.
The discussion above assumes homogeneous attacks, where single type of backdoor is deployed. However, real-world scenarios often involve heterogeneous attacks, where mul- tiple backdoor triggers coexist within the same poisoned model. this setting, the model must simultaneously learn the clean generative process æ + æ and M distinct back- door processes. Formally, the i-th backdoor process is rep- resented as x. + æ. where the poisoned input xF" is obtained by blending the clean input of with a trigger 5; similar to Eq. 2. In our method, different triggers are strate- gically applied to distinct locations (i.e., using different binary masks M,) to differentiate between attacks.
How to deal with such a mixed sample path? Next, we will introduce our method, starting with a naive solution, and then propose the final heterogeneous method.
4.2. A Naive Method for Heterogenous Backdoor Injection
Existing methods often require extensive modifications to both the forward and reverse diffusion processes in denois- ing diffusion models execute backdoor attacks, enabling the model to learn an undesired correlation between the backdoor trigger and the backdoor target. Fortunately, on top of the diffusion bridge model I2SB, we theoretically re- veal that the backdoor attacks towards bridged-basc -ased models can be implemented more easily:
As described in Eq. 1, we prepare pairs of images to train the DSB model. We use D. (( N and D. = { , to denote the clean and i-th poi- soned training datasets, containing N, and N, image pairs respectively. The image pairs for benign generation can be easily obtained. For example, the image pairs for the super-

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Proposition 4.1 (Image generation with pair relationship). Given the image pairs (a/ "x or (arg mi) in the train- ing datasets, the ground-truth sample-path of 12SB always generate images consistent with pairwise relationships with t +++1 and + 0.
where p(ilz) = p(ilar To. æf), where refers a specific member among the clean and backdoored distributions.
Thm.4.2 demonstrates that when a single 12SB model is used to fit the sample paths of both the benign distribu- tion of + z6 and all heterogeneous backdoor distributions क1 + x/ the model tends approximate the geomet- ric average of these distributions. Given the significant disparity between the clean and backdoored distributions, this geometric mean often deviates greatly from most target distributions, leading to performance degradations for both benign and malicious tasks. We conduct a simple experi- ment illustrated in Fig. (right) to highlight the limitation of this, where a single model struggles to generate satisfactory outputs for both clean and poisoned images.
Pr demonstrates that the sample path of 12SB auto- matically connects the input and output distribution for I2I generation. In this sense, we no longer need to design spe- cific SDEs to cater to different backdoor generation tasks. In other words, the backdoor triggers can be embedded by simply training the model with poisoned image pairs. As a naive solution, one only needs to simply poison or add a subset of the training data with malicious triggers. Then, given clean dataset De and various types of poisoned image pairs Di, i E [M], we can employ the following objective function to train a backdoored 12SB model:
What if we integrate the generation ability from multiple models? In the upcoming discussions, we will present a solution in this manner.
Crackdocr (0) = e
where
4.3. Exploring Heterogeneous Backdoor Attacks Beyond a Single Model
2
e
p(c) Be, ajjf e(axf 4; 0)
To achieve the performance balance across different tasks, we propose a divide-and -ne--erggeee strategy borrowing the idea from the Mixture of Experts (MoE) mechanism (Chen et al., 2023b; Ma al., 2018: Du al., 2024). The critical point we first train well well-performing 12SB models tailored to indi- vidual tasks independently and subsequently integrate them into unified MixBridge model. The overall framework of the proposed Mi MixBridge is illustrated in Fig. (left).
0t
= p(t) " 11 c(af' , .)
The key issue with this method is that it neglects the model fitting ability. Although 12SB guarantees a flexible sample path, we also need to consider whether the expressivity of the employed model is sufficient to fit such a sample path. Unfortunately, for a complicated heterogeneous backdoor attack, we find that the sample paths can hardly be fitted by a single 12SB model. In the setting of backdoor attacks, the model is required to simultancously generate a clean output zi and a poisoned malicious output æ. The challenge arises because z8 and of have significantly different dis- tributions, whereas their respective inputs 2 and at differ only by small mask o; leading to mutual interference between different tasks.
Stage 1: Divide. We propose a task-specific warm-up strat- egy to pre-train different tasks independently, where each task corresponds an 12SB model. Specifically, let the clean I2SB model be denoted as co(: and the back- doored 12SB experts be denoted as G(+ :; 0,), E [M], totally M+1 experts. In this stage, each model e, is trained independently with its corresponding paired images to mini- mize Eq. 1. For the benign 12I generation task, the expert ce is trained exclusively on clean paired images D.. Sim- ilarly, the backdoored expert e,, i E [M] is trained on the i-th poisoned training dataset D, This stage ensures that each model excels in its specific generation task without interference from others, while also enhancing the overall diversity of the models.
Below, we present a theoretical analysis of its limitations: Theorem 4.2 (Limitations of using single 12SB model for heterogeneous backdoor attacks). Given arbitrary image Fg. the posterior of a trained 12SB model can be formulated as pe (z( (x)). If we assume Va, Ve,co co(x,, t; O) possesses full column rank?, the posterior is proportional to the Geometric Average of the mixture distribution of all generation i...
Stage 2: Merge. After the dividing stage, we merge all pre-trained experts as a mixture model. This requires en- couraging cooperation among these experts and preserving their individual generative capabilities as much possible. To do this, we introduce learnable expert router r to adap- tively determine the contribution of each model to the final output, represented by w = (we, W), ..., WM) E IR TR(+11 Considering that backdoor attacks are typically triggered by input images, we design the router to distinguish high-level
po ( (D) x II p( 2jolii))
(5)
II is plausible that V a. (x, f; 0) has full column rank, given that the number of parameters in the model significantly exceeds the dimensionality of the image feature space.
5

---

MixBridge: Heterogeneous Image-to-lmage Backdoor Attack through Mixture of Schrödinger Bridges
Table 1: Experimental results for super-resolution and per-task backdoor attacks on the CelebA dataset. The backdoor attack results are reported as the average across all individual backdoor tasks. Note that the purple cell represents the optimal value, and the green cell represents the sub-optimal value. SR.: Super-resolution. Entro. (Steal.): Entropy (Stealthiness).
Super-resolution Evaluation (Benign)
Per-Task Backdoor Average
SR.
Fake Face
Anime NSPW
NSFW
(Benign)
SSIM
MSE (E-02)
CLIP
Entru.
FID
PSNR1
ASRT
(E-02)
(C-02)
(Steal.)
12SB
72.59
27.55
81.72
36.71
58.42
0.00
0.00
132.83
25.64
71.13
27.16
66.63
32.77
0.00
135.82
25.71
71.62
25.25
67.44
32.50
0.00
134.46
25.21
67.37
22.43
65.32
30.10
0.00
Single Model
143.42
25.38
69.30
16.00
73.23
62.98
0.00
158.23
25.18
68. 19
12.68
69.47
$3.69
0.00
159.85
25.33
68, 99.99
11.60
71.54
52.11
0.00
161.35
24.98
66.19
3.05
71.59
60.91
0.00
41.48
27.46
83.35
27.06
69.77
33.17
4c-03
41.20
27.33
81.54
25.10
71.16
33.19
3c-03
42.26
27.29
81,40
22.11
70.80
33.04
8c-03
w/o WRS
61.73
26.72
83.18
13.13
83.03
63.79
3c-03
61.25
25.74
76.00
11.06
74.68
62.63
2c-03
63.51
26.63
83.00
11.74
78.32
60.65
1e-03
71.84
25.68
82.33
1.17
88.85
96.45
7e-03
M.B. (Ours)
60.65
26.43
82.17
27.04
69.60
33.25
0.99
68.12
80.57
25.24
71.05
33.08
0.99
66.59
26.70
80.59
22.08
70.57
33.19
0.99
WRS
80.88
24.61
15.83
66.04
83.85
24.07
71.07
12.48
78.85
65.75
77.21
73.69
81.74
85.88
24.36
70.40
1.13
96.98
backdoor patterns embedded in the input features x;. For generality, wi represents either clean images or any type of poisoned image. Concretely, the router calculates the normalized weights follows:
can adaptively approximate the optimal diffusion trajectory by assigning higher weight to the relevant model.
Weight Reallocation Scheme. However, in practice, we observe that simply merging these multiple experts is in- sufficient. To minimize the reconstruction enror (Eq. I), the router r tends to assign a higher weight to the model specifi- cally pre-trained for particular k-th generation task during the divide stage, effectively setting w³k = 1 while assigning wj = for all j + h (Refer to Sec. 5.4). This behavior undermines the model's stealthiness, users can detect anomalies by inspecting the components of MixBridge.
w = r<=1)= Softmax W F(x;) WTF(F:
+ b),
(6)
where F(-) E Rd is deep neural network (e.g., ResNet (He et al., 2016)) used to extract useful features, d is the feature dimension; W E (M+1) is the learnable transformation matrix, and b E M+++ is the bias term.
Subsequently, the output of the MixBri dge at each step t (t E [0, 1]) is computed as the weighted sum:
To address this issue, we propose Weight Reallocation Scheme (WRS) to prevent the weight assignment from being too far from uniform:
(Lxx (m t; (xixx) = < (z) (2)
M
w e (z;
Lwns = Ew
(8)
M
Intuitively, Eq. 8 encourages the router to assign uni- formly smooth contributions to each expert, enhancing the stealthiness of the Mi xxxxirii dgg model.
where ot is the generated image at time f, and again ca(x;, t; Oz) and (x;,,, t; 0,) are the denoising predictions of the clean and backdoored experts, respectively.
Finally, we optimize the following objective function during the merging stage balance generative performance and backdoor stealthiness:
Equipped with Eq.6 and 7q we proceed to fine-tune MixBridge using all datasets D. and D; (i [M] to rec- oncile performance conflicts between tasks. Ideally, given different inputs (clean or backdoored), Mi xBr xBridge now
C(O = Lanckdoor (fm:x) + ALwRs
(9)
6

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
where 0gix (0,, O1,.,, O,, 0.) represents all trainable parameters included in Mi MixBridge dge, 0z is the learnable parameters of the router r; À is tunable trade-off hyper- parameter. Detailed training and generation procedures are provided in Alg. and Alg. 2, respectively.
recovered image preserves the original image structure.
Specificity: We use MSE, CLIP score (Hessel et al., 2021), 1), Attack Success Rate (ASR) (the ratio of successful attacks to total attacks) and Shannon information Entropy (Diaz et al., 2002; Murdoch, 2013) for backdoor attack evaluation. A low MSE implies the output image is pixel-wise close to the predefined malicious image. For the CLIP score, we apply CLIP model extract the image embeddings and compute the cosine similarity. A high CLIP score signifies the gener- ated image's features resemble those of the malicious image. For the ASR, we consider backdoor attack successful if the CLIP score exceeds a specific threshold. A high ASR indi- cates the model is sensitive to triggers, leading to successful attacks. In addition, we assess the stealthiness of backdoor attacks using Entropy (Entro.), computed with the weight distribution: H(w) w we M W; log w;, A high entropy suggests a uniform weight distribution, en- hancing the anonymity of experts and the stealthiness of backdoor attacks.
5. Experiments
5.1. Experimental Setup
Datasets and Models. We evaluate our proposed backdoor attack method in two normal tasks, super-resolution and image inpainting.
The experiments of super-resolution are conducted on the CelebA dataset (Liu et al., 2015). The images are resized to 128 x 128, and then the images are further corrupted to 32 32 by a pool filter to create low-resolution input images. The model is trained to recover the high-resolution image (128 x 128) from the low-resolution input (32 x 32). The experiments of image inpainting are conducted on the ImageNet 256 x 256 (Deng et al., 2009). We use the
5.3. Main Results
20%-30% freeform masks from (Saharia et al., 2022a) to
corrupt the images for the inpainting task.
Fig. 2 presents an overall performance visualization of the proposed method. The results show that with the joint effect of MoE and Weight Reallocation Scheme, our approach successfully achieves both high-quality generations and a uniform distribution of expert weights.
We prepare M backdoor attacks: Fake Face, NSFW, and Anime NSFW. For each backdoor attack, prepare Nm 10 malicious images for Dm to ensure the diversity of the backdoor attacks. The Fake Face alters an input image to generate a completely different face, while the other two attacks generate pornographic outputs. We manually select ten images of the same person from the CelebA dataset for the Fake Face attack. For the latter two, we randomly select ten NSFW images and ten Anime NSFW images from (Yang et al., 2023a)² and Hugging Face
1. CELEBA DATASET
We present the super-resolution and the per-task backdoor average results on the CelebA dataset in Tab. 1.
Super-resolution. Our findings show that even when the MixBrico idge model is trained on both the super-resolution task and backdoor attacks simultancously, outperforms the baseline 12SB model. For example, the FID of the Mi xiixBrieen dgg model trained with the Fake Face backdoor attack is close to half that of the 12SB model. This suggests that the MixBr i igge model enhances the model's capacity, enabling it to successfully conduct backdoor attacks while also benefiting benign image generation. On the contrary, the single model performs significantly worse than the base- line. This validates the challenge of solving both benign tasks and backdoor attacks within single model.
For fair comparison, we adopt the same model architec- ture and parameter settings as in (Liu et al., 2023). Each expert & in the MixBridge idge is a UNet block pre-trained on ImageNet 256 x 256 (Dhariwal & Nichol, 2021). The router r is implemented
ResNet50 model.
5.2. Evaluation Metrics
We evaluate our backdoor attack method from two perspec- tives, utility and specificity.
We use FID (Heusel et al., 2017), PSNR, and SSIM (Wang et al., 2004) to evaluate the utility of super- resolution and image inpainting tasks. A low FID indicates that the output image closely matches the input data dis- tribution. The high PSNR and SSIM values indicate the
Utility:
Backdoor Attacks. As for the average performance of back- door attacks, the results indicate our method successfully executes heterogeneous backdoor attacks, outperforming the single model in generating higher-quality malicious im- ages. Notably, the entropy of the weight distribution in the Mix MixBridge trained with WRS is significantly higher than in the model without WRS, which implies WRS enhances the stealthiness of the Mi xBri dge by promoting more uni- form weight distributions. It is also worth emphasizing that
httpp:://g coub.com/aaaa...cc w1w0000x0000kkk scraper hettps:/hhugiingfeee wtt.:/cc...odatattttttt Boostewin/reeashed-- nsw--ful
7

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Super-nesolotion (Benign)
Inpainting (Benign)
Fake Face
Anime NSPW
Fakse Face
NSFW
Aniine NSFW
Weight Aug.
F1
MixBridge
MixBridge tw/o WRS)
Single Model
Stealthiness
CelebA
ImageNet
Figure 2: Visualization of the generation results of the MixBridge. We visualize the results of different generation tasks with different methods. Clearly, our Mi xBridge achieves high performance across all tasks. Additionally, reorganize the weight values in descending order and present the average weight distribution in the "Weight Average" column. The results demonstrate that, with the help of the Weight Reallocation Scheme (WRS), we encourage more uniform distribution of the weights, thereby enhancing the stcalthiness of the model.
2.0
100
1.5
96
95
1.0
Wag
D Ielo wag
1.0
D waS
0.5
Nies.
O Kies.
85
0.5
+ Mius
0.0
0.0
Falu Face
NSFW
Anime NSPW
88
Fales Face NSPW
Anime NSFW
80
Fake Face
NSFW
Anime NSFW
Falue Face
NSFW
Anime NSFW
(a) MSE
(b) CLIP Score
(c) ASR
(d) Entropy
Figure 3: Results of the backdoor attacks on the ImageNet. The results are evaluated by models trained with four tasks, image inpainting, Fake Face, NSFW, and Anime NSFW.
the model trained with WRS generates better malicious im- ages. This may be attributed to the fact that, compared to super-resolution tasks, the target distributions for the back- door attacks are more distinct from the input distribution. Thus, without WRS, the model tends to prioritize the easier However, when trained with WRS, the model is encouraged to focus more on the backdoor attack tasks, resulting in better performance in generating malicious images.
100
20
75
50
25
5
task losk-super-resoluude
0.94
0.96
0.98 1.00
0.46
0.50
0.54
0.58
(a) Without WRS
(b) With WRS
Figure 4: The distribution of weight w. The weight con- centrates around for the backdoor attack without WRS (Left), and the weight balances 0.5 with WRS (Right).
5.3.2. IMAGENET DATASET
Due to the space constraint, the detailed numerical results of are provided Appendix D.3. Here we show summa- rized backdoor attack results in Fig.3. demonstrates that Mi iiixBriide dge achieves nearly 100% ASR while generating hugher-quality images compared to single model.
the model, and we record the weight w1 assigned to the expert Ej. The results are shown in Fig. 4. Unsurprisingly the weights wi concentrated around indicating that the MixBri model heavily relies on the expert pre-trained in backdoor generation task. In contrast, if we apply WRS during training, the weights are balanced around 0.5.
5.4. Effect of Weight Reallocation Scheme
We construct simple Mi xBriider idge model consisting of e and c1. A batch of 128 poisoned images xf is input into
The weight distribution directly impacts the stcalthiness of the model. Without WRS, 61, pre-trained for backdoor
8

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
generation, tends generate images with backdoor-related features even for clean inputs. Conversely, with WRS, <1 is forced to contribute more to the benign image generation, thus 61 has to generate outputs devoid of malicious features.
Acknowledgement
This work was supported in part by the Fundamental Research Funds for the Central Universities, in part by the National Key R&D Program of China under Grant 2018AAA0102000, 2024QY210004 and 2022YFC3302300, in part by National Natural Science Foundation of China: 62236008, 62441232, U2IB2038, U23B2051, 62122075, 62206264 and 92370102, in part by Youth Innovation Pro- motion Association CAS, in part by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDB0680201, and in part by the Postdoctoral Fellowship Program of CPSF under Grant G2B20240729.
5.5. Additional Results
Due to the space constraints, we provide additional exper- iment results in Appendix D. In particular, the detailed re- sults and analysis concerning the CelebA and ImageNet datasets are presented in Appendix D.2 and Appendix D.3. In addition, we visualize the outputs of each expert in the MixBridge model in Appendix D.4 to demonstrate the effects of WRS. Besides, we discuss the performance of the MixBridge in different poison rates in Appendix D.5.
References
An, S., Chou, S.-Y., Zhang, K., Xu, Q.. Tao, G., Shen, G., Cheng, S., Ma, S., Chen, P.-Y., Ho, T.-Y., et al. Elijah: Eliminating backdoors injected in diffusion models via distribution shift. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10847-10855, 2024.
6. Conclusion
In this paper, we introduce a novel diffusion Schrödinger Bridge model (DSB), Mi xBri xgri to investigate the het- erogeneous backdoor attacks in Image-to-Image (I2I) gen- eration tasks. To mitigate the interference between differ- ent generation tasks, we incorporate the Mixture of Ex- perts (MoE) architecture and propose a divide-and-merge training strategy, which enables the MixBri dge to gener- ate high-quality benign images when the inputs are clean, and heterogeneous malicious high-quality outputs when the inputs are poisoned. Extensive experiments on two datasets demonstrate the versatility and harmfulness of the MixBridge model. As a first step in this direction, we present MixBrix i dge as a red-team tool to better understand the vulnerabilities of diffusion bridge models and to inspire further research on defensive methods against backdoor attacks on 12I generation tasks.
Bao, S., Xu, Q., Yang, Z., He, Y., Cao, X., and Huang, Q. Aucpro: Auc-oriented provable robustness learning. IEEE Transactions on Pattern Analysis and Machine In- telligence, 2025.
Chao, C.-H., Sun, W.-F., Cheng, B.- Lo, Y.-C., Chang, C.-C,, Liu, Y.LL., Chang, Y.-L., Chen, C.-P., and Lee, C.- Y. Denoising likelihood score matching for conditional score-based data generation. In International Conference on Learning Representations, 2021.
Chen, W., Song, D.,, and Li, B. Trojdiff: Trojan attacks on diffusion models with diverse targets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4035-4044, 2023a.
Impact Statement
The growing threat of backdoor attacks against diffusion models has garnered increasing attention. However, back- door attacks targeting bridge-based diffusion models re- main an unexplored area. In this paper, we aim to fill this gap by investigating heterogeneous backdoor attacks on the MixBridge model. It crucial note that we have mo- saicked the sensitive regions in the visualization results in this paper to minimize any potential harm to the readers. All the pomoggappicc contents used in this study are sourced from public datasets. While there is a risk that the proposed method could be misused for unethical purposes, we believe that this research serves more important purpose as tool to understand the mechanisms behind backdoor vulnerabili- ties in bridge-basee based diffusion models. Moreover, we hope this work will inspire the research community to prioritize the development of effective defense strategies against such attacks.
Chen, Y., Georgiou, T. T., and Pavon, M. Stochastic control liaisons: Richard sinkhorn meets gaspard monge on a schrodinger bridge. Siam Review, 63(2):249-313, 2021.
Chen, Z., Shen, Y., Ding, M., Chen, Z., Zhao, H., Learned- Miller, F. G., and Gan, C. Mod-squad: Designing mix- tures of experts as modular multi-task learners. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 11828 118377 2023b.
Chou, S.-Y., Chen, P.-Y,, and Ho, T.-Y. How to backdoor diffusion models? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4015-4024, 2023.
Chou, S.-Y., Chen, P.-Y,, and Ho, T.-Y. Villandiffusion: A unified backdoor attack framework for diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
9

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
De Bortoli, V., Thornton, J., Heng. J., and Doucet, A. Diffu- sion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34:17695-1777 17709, 2021.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Deng, J. Dong. W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 245. Icce, 2009.
Ho, and Salimans, T. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Huang, R., Lam, M., Wang, J., Su, D., Yu, D., Ren, Y., and
Zhao, Z. Fastdiff: A fast conditional diffusion model for high-quality speech synthesis. In IJCAI International Joint Conference on Artificial Intelligence, pp. 4157- 4163. IJCAI: International Joint Conferences on Artificial Intelligence Organization, 2022.
Diaz, C., Seys, S., Claessens, J., and Preneel, B. Towards measuring anonymity. In International Workshop on Privacy Enhancing Technologies, pp. 54-68. Springer, 2002.
Doan, K., Lao, Y., Zhao, W., and Li, Lira: Learnable, im- perceptible and robust backdoor attacks. In Proceedings of the IEEE/CVF international conference computer vision, pp. 11 11966-1197,, 2021.
Huang, Y., Guo, Q., and Juefei-Xu, F. Zero-day backdoor attack against text-to-image diffusion models via person- alization. arXiv preprint arXiv:2305.1070. 1(2), 2023.
Jiang, W., Li, H., He, J., Zhang, R., Xu, G., Zhang, T., and Lu, R. Backdoor attacks against image-to-image networks. CoRR, 2024.
Du, Y., Zhao, S., Zhao, D., Ma, M., Chen, Y., Huo, L., Yang, Q. Xu, D., and Qin, B. Mogu: A framework for enhancing safety of open-soureed Ilms while preserving their usability. arXiv preprint arXiv:2405. 14488, 2024.
Kong, Z.,, Ping, W., Huang. J., Zhao K., and Catanzaro, Diffwave: A versatile diffusion model for audio synthe- sis. In International Conference on Learning Representa- tions.
Edition, F., Papoulis, A., and Pillai, S. U. Probability, ran- dom variables, and stochastic processes. McGraw-Hill Europe: New York, NY, USA, 2002.
Léonard, C. A survey of the schrödinger problem and some of its connections with optimal transport. Discrete & Con- timuous Dynamical Systems-A, 1533-115331, 2014.
Gu, T., Dolan,Gavitt, B., and Garg, S. Badnets: Identify- ing vulnerabilities in the machine learning model supply chain, arXiv preprint arXiv: Tttv:/700 7177000877333 2017.
Liu, G.-H., Vahdat, A., Huang, D.-A., Theodorou, E.,
Nie, W., and Anandkumar, A. 12sh: Image-to-image schrödinger bridge. In International Conference on Ma- chine Learning, pp. 00042-22062. PMLR, 2023.
Guan, Z., Hu, M., Li, S., and Vullikanti, A. Ufid: A unified framework for input-level backdoor detection on diffu- sion models. arXiv preprint arXiv:2404.01100 2024.
Liu, Z., Luo, P., Wang, X., and Tang. X. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Gushchin, N., Kolesov, A., Korotin, A., Vetrov, D. P., and Burnaev, Entropic neural optimal transport via diffu- sion processes. Advances in Neural Information Process- ing Systems, 36, 2024.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference Learning Representations.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Lou, A., Meng, C., and Ermon, Discrete diffusion mod- eling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learn- ing.
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. Clipscore: A reference-fre freee evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, 2021.
Ma, J., Zhao, Z., Yi, X., Chen, J., Hong, L., and Chi, E. H. Modeling task relationships in multi-task learning with multi-gate mixture-of-expprts. Proceedings of the 24th
10

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1930-1939, 2018.
Symposium Security and Privacy (SP), pp. 807-825. IEEE, 2024.
Meng, B., Xu, Q., Wang. Z., Cao, X., and Huang. Q. Not all diffusion model activations have been evaluated discriminative features. Advances in Neural Information Processing Systems, 37:55141 111551775777, 2024.
Shi, Y., De Bortoli, V., Campbell, A., and Doucet, A. Diffu- sion schrödinger bridge matching. Advances in Neural Information Processing Systems, 36, 2024.
Song, J., Meng, C., and Ermon, Denoising diffusion implicit models, In International Conference on Learning Representations,
Mo, Y., Huang. H., Li, M., Li, A., and Wang, Y. Terd: A unified framework for safeguarding diffusion models against backdoors. In International Conference on Ma- chine Learning, pp. 35892-35909. PMLR, 2024.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole. B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, b.
Murdoch, J. Quantifying and measuring anonymity. In International Workshop on Data Privacy Management, pp. 3-13. Springer, 2013.
Struppek, L., Hintersdorf, D., and Kersting, K. Rickrolling the artist: Injecting backdoors into text encoders for text- to-image synthesis. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4584- 4596, 2023.
Nelson, E. Dynamical theories of Brownian motion, volume 101. Princeton university press, 2020.
Nichol, A. Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Mcgrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and edit- ing with text-guided diffusion models. In International Conference on Machine Learning, pp. 16784-16804. PMLR, 2022.
Tang, K.-T. Mathematical methods for engineers and scien- tists, volume 1. Springer, 2007.
Tong, A., FATRAS, K., Malkin, N., Huguet, G., Zhang, Y., Rector Rector-Brooks, J., Wolf, G., and Bengio, Y. Improv- ing and generalizing flow-based generative models with minibatch optimal transport. Transactions on Machine Learning Research, 2024.
Pan, Z., Yao, Y., Liu, G., Shen, B., Zhao, H. V., Kompella, R. R., and Liu, S. From trojan horses to castle walls: Unveiling bilateral backdoor effects in diffusion mod- els. In NeurlPS 2023 Workshop on Backdoors in Deep Learning- The Good, the Bad, and the Ugly, 2023.
Wang, H., Shen, Q. Tong, Y., Zhang, Y., and Kawaguchi, K. The stronger the diffusion model, the easier the backdoor: Data poisoning to induce copyright breacheswithout ad- justing finetuning pipeline. In International Conference on Machine Learning, pp. 5 51465-51483, PMLR, 2024a.
Rawat, A., Levacher, K., and Sinn, M. The devil in the gan: backdoor attacks and defenses in deep generative models. In European Symposium on Research in Com- puter Security, pp. 776-783. Springer, 2022.
Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., and Norouzi, M. Palette: image-to-image diffuuson models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022a.
Wang. Y., Yoon. S., Jin, P., Tivnan, M., Chen, Z., Hu, R., Zhang, L.,, Chen, 2., Li, Q., and Wu, D. Implicit image- to-image schrodinger bridge for ct super-resolution and denoising. arXiv preprint (XXi:2443300666 2024b.
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., and Norouzi, M. Image super-resolution via iterative refinement. IEEE Iransactions on pattern analysis and machine intelligence, (344):471334726 2022b.
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E, P. Image quality assessment: from error visibility to struc- tural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
Sasaki, H., Willcocks, C. G., and Breckon, T. P. Unit-ddpm: Unpaired image translation with denoising diffusion prob- abilistic models. arXiv preprint arXiv:2104 10.05558, 2021.
Wang. Z., Zhang, J., Shan, S., and Chen, X. T2ishield: Defending against backdoors on text-to-image diffusion models. In 18th European Conference on Computer Vi- sion, ECCV 2024, pp. 107-124. Springer Science and Business Media Deutschland GmbH, 2025.
Schrödinger, F. Sur la théorie relativiste de l'électron et I'interprétation de la mécanique quantique. In Annales de l'institut Henri Poincaré, volume pp. 269-310, 1932.
Wen, J., Deng, S., Fei, L., Zhang, Z., Zhang, B., Zhang, Z., and Xu, Y. Discriminative regression with adaptive graph diffusion. IEEE Transactions on Neural Networks and Learning Systems, 35(2)) 2022.
Shan, S., Ding, W., Passananti, J., Wu, S., Zheng, H.,, and
Zhao, B. Y. Nightshade: Prompt-specific poisoning at- tacks on sot-to-imaae generative models. In 2024 IEEE

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Yang, Z.,, Xu, Q. Bao, S., He, Y., Cao, X., and Huang. Q- When all need is piece of the pie: A generic framework for optimizing two-way partial auc. In Inter- national Conference on Machine Learning, pp. 11820- 11829. PMLR, 2021.
Yang, 7.,, Xu, Q., Bao, S., He, Y., Cao, X., and Huang, Q. Optimizing two-way partial auc with end-to-end framework. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):10228-10244 2022.
Yang, Z., Xu, Q., Bao, S., Wen, P., He, Y., Cao, X., and Huang, Q. Auc-oriented domain adaptation: from theory to algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023a.
Yang, Z., Xu, Q., Hou, W., Bao, S., He, Y., Cao, X., and Huang. Q. Revisiting auc-oriented adversarial training with loss-agnostic perturbations. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(12):15494- 15511, 2023b.
Ye, J. C., Kim, B., KWON, G., et al. Unpaired image-to- image translation via neural schrödinger bridge. In The Twelfth International Conference on Learning Represen- tations, ICLR 2024. International Conference on Learning Representations, 2024.
Zhai, S., Dong, Y., Shen, Q. Pu, S., Fang, Y., and Su, H. Text-to-image diffusion models can be easily backdoored through multimodal data poisoning. In Proceedings of the 3 ACM International Conference on Multimedia, pp. 1577-1587, 2023.
Zhang, Y., Jia, J., Chen, X., Chen, A.. Zhang, Y., Liu, J., Ding, K., and Liu, S. To generate or not? safety-driven un- learned diffusion models are still easy to generate unsafe images... for now. In Europeun Conference on Computer Vision, pp. 385-403, Springer, 2025.
Zhou, L., Lou, A., Khanna, S., and Ermon, S. Denoising diffusion bridge models. In The Twelfth International Conference on Learning Representations.
12

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Appendix
Table of Contents
Schrödinger Bridge and 12SB
14
Proof
15
Algorithm Expression
17
Additional Experiment Results
17
D.I
Implementation Details
17
D.1.1 Dataset
17
D.1.2 Baseline
18
D.1.3 Implementation Details
18
D.2 Backdoor Attack on the CelebA Dataset
19
D.3 Results the ImageNet Dataset
20
D.4 Analysis of the Weight Reallocation Scheme
20
D.5 MixBridge with Different Poison Rate
20
D.6 MixBridge with Different Trigger Size
20
D.7 Backdoor Attack Defense
21
D.8 Additional Visualization Results
22
13

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
A. Schrödinger Bridge and I2SB
The Schrödinger Bridge (Schrödinger, 1932; Léonard, 2014) operates by achieving an eetropy-regullrized optimal transport between two given distributions. This unique property allows the Diffusion Schrödinger Bridge (DSB) to directly transfer images between distinct distributions, which can address a wide range of I2I tasks, such as super-resolution, inpainting, and deblurring (Gushchin et al., 2024; Wang al., 2024b; Liu et al., 2023).
Given two marginals PA and pB, the DSB problem can be formulated by minimization problem over a stochastic process Py:
min
DKL(P
Pg Pa
where Q represents a reference process such as Brownian motion.
If we characterize the DSB problem with SDE form (Léonard, 2014; Chen et al., 2021), obtain the forward and backward SDEs conditioned by the marginals at zo PA and æ1
PBi
dæ. (fxx) 3 log (at, d) + B,dW,
(10)
dæ. [.xx) 3, V log v/ (z,, ()] + v B,dW,, t.
where / is a drift function, 3e is a diffusion coefficient, and W, is a Wiener process. In addition, v and い satisfy the Kolmogorov backward//forward equations:
a (2), t)
1 (v..t.))
= f(x))
+ 2²
Ot
dx
a ( t)
0²
(1 (xx))
+
B3 // (zz, t)].
Ōt
0x.
In 12SB (Liu et al., 2023), f(x is set to zero, then v and v can also be regarded as path p.d.f functions for the following two SDEs respectively:
dx BidW., xo (1( 0),
(1I)
dæ = Va,WW,
(-)
Under this special case, the posterior of given the paired image (æo. x1) enjoys closed-form solution according to the Nelson's 's duality (Nelson, 2020):
((x, FD, x1) = N (x/; Me, Σ.)
of
μ4 कī of
20 +
o? + o
E1.
(12)
0,
Σ = of + o
T
where
[" B,
03
(13)
o, L. Brdr
(14)
represent the variance of the SDE time in the diffusion process. Therefore, 2; be expressed as linear combination of and æ1
In 12SB, the sampling path is fitted on top of the standard diffusion model based on the results above. The corresponding objective function expressed as follows:
((0) = 1e
(1 (xx)) - Van (ogp(()()=))))))
(15)
where c(x,, 0) s0,))) and V logp(a, (20) V Voog (( t).
14

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
B. Proof
Proposition 4.1 (Image generation with pair relationship). Given the image pairs (x) xx) or (ar 2f) in the training datasets, the ground-trune cample-patl of I2SB always generate images consistent with pairwise relationships with 1 and O.
Proof of Proposition 4.1. As discussed in Eq. 14, the limiting variances time 0, L can be expressed as follows:
lim of lim [ 8,dr = 0, 100 t-0
lim o = lim
8, = 0.
t+1
Thus, one can get the limiting mean and variance for the 12SB bridge follows (see Eq. for the mixture distribution):
lim Pr lim xy [-0
20.
1-0
lim Σ, =0. 100
lim 14 = lim x; = 31,
[-1
1-+1
lim Σ, = O. f-1
Here, (zog x1) represents the images with pairwise relationships in our framework, which can be either clean pairs (ag azft)) or poisoned pairs (zf, 2f). According to the continuity of the guassian p.d.f function w.r.t its mean and variance, we reach the limiting distribution of o 2o. 31:
lim ( Ro 21) = O0n t-+0
lim (x DD 21) 82y1
where 6g is Dirac's delta function such
0, x + u
(z)
(16)
00, x = u
Assume p(zp, 21) be the marginal density of the input-output pair, have:
= lim / q( x1) p( dægdæ
lim g(x,) (+0
= ((
= p(xo).
where the second equality follows that that if q æo æ1) + 8z, in distribution, then f(x,)) y) * q( z x1) f (2o. 8)
almost everywhere for any bounded, almost everywhere continuous, compactly supported f (Tong et al., 2024).
Similarly,
lim g(x;) = p(x1). [+
which guarantees that the generated images align with the pairwise relationship in the training dataset.
Theorem 4.2 (Training naive model for heterogencous backdoor attacks). Given arbitrary image Lo. the posterior of trained 12SB model can be formulated as Po (x,( If we assume V.. (I:; t; 0) possesses full column ranké, the
'It is plausible that Va, c 1; 0) has full column rank, given that the number of parameters in the model significantly exceeds the dimensionality image feature space.
15

---

MixBridge: Heterogeneous Image-to-lmage Backdoor Attack through Mixture of Schrödinger Bridges
posterior is proportional the Geometric Averuge of the mixture distribution all generation iasks,i.e.;
(a (2)) II pla, (o, s)p(i22),
(17)
where p(]|=) p(i) vo, vo, ai), where refers to a specific member among the clean and backdoored distributions.
Proof of Theorem 4.2. Given so p(zo): p(()) defined in Eq. 3, æn and æ belong to mixture of clean images and malicious images. As discussed in Appendix A, Eq. 4 can be expressed by Eq. 15:
Naive (0) = E
p(c) E Ex, io affle (Ils0(=;)) Vag log (( c)l
p(()) (Rr.ee "af af (1)
on log (2t (=.g)))))))))))) :)(p))
For simplicity, we use to denote the i-th generation task for the model, including benign image generation and malicious we reach the following equivalent formulation:
backdoor attacks. Moreover, let LE (pp())))
Naive ( Ep(t)
p(i) Bep()))) (Ils((2;) Var poz (20. 1)
To optimize the objective function, the derivative V Naviee (0) should 0.
V Va, Native ( Ep(i)
p(i) p(zli) (2(s0(x;; Va. logp(x, i)) Va, s2())))
2V n, s0(s)) E((z,) (so(x):) Va, 1ogp(xr(200,) i))
0.
Therefore, if we assume Ve, so(x)) possesses full column rank, we obtain:
s0(x;) = Eir(z,0 Va, log pogg(ar/ i)
,(,
According to the Bayes Theorem for probability density functions (Edition et al., 2002),
p(2,1) = =p((lz). {(,
Thus,
so(2x)) = Σ p(llz) Va, log por
= Va. log
Il p( [xp, z)p(llz).
Now if we assume that sæ(x;) can also be written as the score of distribution (xx (z0), obtain:
se(x:)
Va, log I p t)p(il=) = Va. log (2
16

---

MixBridge: Heterogeneous Image-to-lmage Backdoor Attack through Mixture of Schrödinger Bridges
According to the Fundamental Theorem for line integrals (Tang, 2007), we integrate the equation for both sides:
J
/
Ve, log
I p( (o i) dc
Va, log pe x 20)
c(
where c(ō - æzi) is a curve connecting an arbitrary anchor point @ to ær. We obtain:
LHS log II p( zo. i) C1,
RHS log po (z, xD) C2.
Therefore, s0(x;) fits distribution that is proportional to the Geometric Average of the mixture of clean and poisoned distribution.
Po ( xv) II p(a, a æ d)p(1)z),
C. Algorithm Expression
In this section, we provide detailed explanation of our method's training and image generation processes, as outlined in Alg. and Alg. 2.
Algorithm MixBri dgge Training
Input: Image pairs datasets D. and D;, E N] repeat
2: Sample ~ ((( 1])
3: Sample (w 2z) from D.
4: Generate af ~ 0(0ff=26 2 by Eq. 12
5: Train the expert ce(s, : 0z) by Eq. 1
6: until The expert C. converges
7: for each i E [1, do
8: repeat
9: Sample (10 1])
Sample (x/) p. from Dj
11: Generate a."" (x la a p. by Eq. 12
Train the i-th expert a 0,) by Eq.
13: until The expert 6 converges
14: end for
15: Construct the Mi MixBridge model (-
16: repeat
17: Sample ~ [[([[[1 1])
18: Sample (zt, æi) from D. and D;
Generate ot (xx [zj xij by Eq. 12
Train the MixBridge 6xix(" (-x; OMLx) by Eq. 9
21: until The MixBridge model (xix converges
D. Additional Experiment Results
D.I. Implementation Details
D. DATASET
We evaluate our xBri dge on the following benchmark datasets.
17

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Algorithm 2 MixBrrion dge Generation
Input: arbitrary imput æं p(æ1), trained MixBridge age model (x 0mix)
1: for N to 1 do
2: Predict m by Eq 7 and Eq. 1
3: Sample - ~ p(= -0 a
4: end for
5: Return zi
CelebA (Liu et al., 2015); CelebFaces Attributes Dataset (CelebA) is a large dataset containing over 200k celebrity images labeled with various facial attributes. The images are annotated with identity information, which is ideal for executing the Fake Face backdoor attack. On this dataset, we conduct the benign super-resolution generation and three heterogeneous backdoor attacks, including Fake Face, NSFW, and Anime NSFW.
ImageNet (Deng et al., 2009): ImageNet is a visual dataset containing more than 14 million images spanning a wide range of categories and scenarios. The large-scale dataset provides a comprehensive evaluation of the effectiveness of our Mi xBr igge On this dataset, we conduct the benign image inpainting generation and the three heterogeneous backdoor attacks.
D.1.2. BASELINE
Since no prior studies have focused on backdoor attacks in the 121 bridge-based diffusion model, we utilize the 12SB (Liu et al., 2023), a Schrödinger Bridge model, as the baseline method. We evaluate the performance of the 12SB model trained solely for the benign generation tasks. In addition, we also evaluate single 12SB model trained for the benign generation tasks and heterogeneous backdoor attacks as discussed in Sec. 4.2. We compare our MixBri idge with the two baselines in terms of the benign generation tasks and the heterogeneous backdoor attacks below, demonstrating that the Mi xBr iBgidge generates high-quality benign images and effectively performs heterogeneous backdoor attacks.
D.1.3. IMPLEMENTATION DETAILS
Our MixBrice consists of a series of DSB experts and a router to be trained. Each expert is a U-net block, and the router is a ResNet50 model.
For the CelebA dataset, we resize the raw image into 128 x 128, and further downsample the images into 32 x 32. The model takes the low resolution inputs and generates benign malicious images with high resolution. In this case, each expert contains 105.2M parameters.
For the ImageNet dataset, we randomly corrupt the 256 x 256 images with 20% - 30% freeform masks from (Saharia et al., 2022b). The model takes the corrupted image and generates restored outputs or malicious images. In this case, each expert is pre-trained on ImageNet (Dhariwal & Nichol, 2021), containing 552.8M parameters.
We develop a divide-and aedle-and-mone merge strategy to train the MixBridge and follow the settings in I2SB (Liu et al., 2023). Specifically, set the learning rate to 5 10-5 with AdamW optimizer (Loshchilov & Hutter) in both stages. We adopt 1000 training intervals (i.e., steps between f and = 0), with the diffusion variance increasing linearly from 10-4 to 2 x 10-2, In the first stage, we train each expert for 2500 iterations using single 3090 24GB GPU. In the second stage, we employ model parallelization, assigning experts to different 3090 24GB GPUs. The combined MixBrin i igge model is trained for 1000 iterations. In each iteration, we train a batch of 256 image pairs. The portions of different image pairs are set to be equal.
As for the router, we train the router with the same learning rate and optimizer. In the first stage, we train the router for 15000 iterations, each iteration contains batch of 256 images. In the second stage, the router is integrated into the MixBridge model and trained alongside other experts.
We use FID (Heusel et al., 2017), PSNR, and SSIM (Wang et al., 2004) to evaluate the utility of super-resolution and image inpainting tasks. These metrics are calculated by comparing the generated benign images with their corresponding ground-trui images.
As for the backdoor attack, we use MSE, CLIP score (Hessel et al., 2021), the Attack Success Rate (ASR) (the ratio of
18

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
successful attacks total attacks), and the Shannon information Entropy (Diaz al., 2002: Murdoch, 2013) evaluate backdoor attack performance. The MSE and CLIP predefined backdoor image. A backdoor attack is considered successful if the CLIP score exceeds a predefined threshold, which we set at 0.7 cosine similarity. The ASR is computed as the ratio of successful attacks to the total number of validation images. Additionally, the Entropy, computed based on the weights experts, evaluates the anonymity of these experts. Higher entropy indicates a more uniform weight distribution, leading to a better stealthiness of the MixBridge.
assess the similarity between the generated backdoor image and the
D.2. Backdoor Attack on the CelebA Dataset
We present the detailed numerical results of Fake Face, NSFW, and Anime NSFW backdoor attacks in Tab. 6, Tab. 7 and Tab. 8, respectively. We summarize the backdoor attack performance in Fig. which clearly shows that the MixBr idge achieves nearly a 100% success rate in executing backdoor attacks, while also generating higher-quality backdoor images compared to the single I2SB model.
100
100
2.0
Nodd
90
O Segl
Negla
1.5
O SiggeWesd
80
80
+ Madicoll (wll
www.con/swwgg
1.0
+ Mawkkook
Mids
O Nad:
60
Mad.
0.5
O Made
Eake
NSFW
Anime NSFW
NSFW
Anime NSFW
Fake Face
NSEW
Anime NSFW
Fake NSFW
Anime NSFW
(a) MSE
(b) CLIP Score
(c) ASR
(d) Entropy
Figure 5: Results of the backdoor attacks on the CelebA. The results are evaluated by models trained with four tasks, image inpainting, Fake Face, NSFW, and Anime NSFW.
Table 2: Experimental results for image inpainting and per-task backdoor attacks on the ImageNet dataset. The backdoor attack results are reported as the average across all individual backdoor tasks. Note that the purple cell represents optimal value, and the green cell represents the sub-optimal value. IP.: Image Inpainting. Entro. (Steal.): Entropy (Stealthiness).
Image Inpainting Evaluation (Benign)
Pask Backdoor Average
IP.
Fake
Anime NSFW
NSFW
(Benign)
Face
SSIM
MSE (E-02)
CLIP
Entro.
FIDA
PSNR1
ASR1
(E-02)
(F-02)
(Steal.)
12SB
14.13
25.21
87.17
42.03
44.52
0.00
23.51
23.34
84,98
29.38
60.94
30.00
23.99
23.85
85.68
28.88
33.33
24.45
23.14
84.74
24.59
33.33
Single Model
30.51
84.33
62.75
28.27
23.16
84.72
12.83
77.60
63.30
28.20
23.17
84.75
12.12
79.43
66.63
29.62
22.92
84.38
1.19
93.60
89.88
14.06
25.12
87.12
29.58
61.17
33.24
8c-03
15.58
25.05
86.75
28.37
61,20
33.33
8c-03
14.60
25.08
86.56
24.12
62,36
33.27
9c-03
w/ WRS
18.65
24.73
86.79
17.59
78.05
66.25
3e-02
18.11
24.90
86.42
12.74
79.15
66.26
4e-02
18.20
24.81
86.76
12.04
79.46
66.67
3e-02
18.73
24.93
86.95
0.55
95.85
98.97
9c-02
M.B. (Ours)
16.26
25.12
87.05
29.53
62.43
33.33
0.99
17.18
25.16
87.13
28.56
61.37
33.33
1.00
16.86
25.18
87.20
24.55
62.33
33.33
1.00
WRS
17.65
86.68
17.22
78.43
66.22
1.58
17.62
24.95
86.70
12.72
79.55
66.47
1.58
17.44
24.87
86.65
11.88
79.43
66.31
1.58
18.26
24.82
86.89
0.59
95.51
99.31
1.99
19

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
D.3. Results on the ImageNet Dataset
We present the image inpainting and the per-task backdoor average results on the ImageNet dataset in Tab. 2.
Image Inpainting. For the benign image inpainting task, the results demonstrate that the Mi MixBridge model successfully achieves performance comparable to the baseline 12SB model. However, the baseline 12SB model's performance is slightly better. This outcome can be attributed to the nature of the image inpainting task, which focuses on generating the corrupted regions of the image (typically 20% 30%) while the rest of the image is intact and directly corresponds to the ground-truth. As a result, the performance of 12SB is quite strong and difficult to surpass. In contrast, the single model performs significantly worse, which demonstrates the effectiveness of our Mi idge model.
Backdoor Attacks. Tab. 2 presents the average performance of the heterogeneous backdoor attacks. We also present the detailed numerical results of Fake Face, NSFW, and Anime NSFW backdoor attacks in Tab. 9, Tab. 10, and Tab. 11, respectively. According to the numeric results, the Mi xBr mixxridde idge generates higher-quality malicious images with higher ASR compared to the single model, which demonstrates that the MixBr idge model successfully enables heterogeneous backdoor attacks. Moreover, the entropy of the weight distribution in the Mi xBr idg trained with WRS is much higher, which suggests that WRS encourages weights to distribute more uniformly, enhancing the anonymity of experts and the stealthiness of the Mi xBridge.
Extensive visualization results in Appendix D.8 further highlight that the Mi milxBridge restores the corrupted images with fine details for the image inpainting task and generates malicious images with superior quality. In contrast, the single model suffers from conflicts between different generation tasks, resulting in poor restorations for the image inpainting task and vague details in backdoor attack outputs.
D.4. Analysis of the Weight Reallocation Scheme
We analyze the effect of the Weight Reallocation Scheme (WRS) by visualizing the outputs of each expert in the Mi xBr i dige model. In particular, we compare the model with and without WRS, trained on the super-resolution task and three heterogeneous backdoor attacks (i.e., Fake Face, NSFW, and Anime NSFW) using four experts. The visualization results are shown in Fig. 6. For the model trained without WRS, when performs the super-resolution task, the outputs of eg, 63, and e, retain certain raclttd features. contrast, when WRS applied, all experts generate images with no relation to the backdoor images. These results demonstrate that WRS enhances the stealthiness of the model.
In addition, we compare the visualization results among the Mi MixBridge, the Mi xBridge without WRS, and the single model in Fig. 7. Obviously, both the Mi xixBrin i dge models (with and without WRS) generate high-quality images for both benign image generation and heterogeneous backdoor attacks, outperforming the single model. However, with WRS, the average weight distribution of the MixBridge becomes uniform, contributing to more stealthy backdoor attacks.
D.5. MixBridge with Different Poison Rate
In all of our experiments in this paper, each generation task shares the same poison rate with others (i.e., p(c) = =p(),, E [M]) Following (Chou et al., 2023), we evaluate the performance of Mi 1xxxiiige dge under different poison rates (i.e., the proportion of poisoned images in the training dataset). Specifically, we analyze the poison rate in Mi xBridge model consisting of two experts for the benign super-resolution and Fake Face backdoor attack tasks.
Tab. 3 shows the results of the Mi MixBridge model for varying poison rates. For the benign super- resouuieso task, lower poison rate leads to better performance. Notably, except for the model trained with dataset containing 90% poisoned images, the MixBridge igg performs relatively well in other settings.
On the contrary, the backdoor attack tasks seem to be not sensitive to the poison rate. Even if the poison rate is only 10%, the model still achieves a 98.30% ASR.
Therefore, we draw a conclusion that the MixBridge achieves both utility and specificity with even an extremely low poison rate, demonstrating the robustness of MixBridge.
D.6. MixBridge with Different Trigger Size
In previous experiments, we set the trigger size 32 x 32 in a 128 x 128 image. Intuitively, the trigger size highly relates to the stealthiness of the backdoor attacks. If the trigger is large, it can be easily detected in real scenarios. Here, conduct
20

---

MixBridge: Heterogeneous Image-to-lmage Backdoor Attack through Mixture of Schrödinger Bridges
Table 3: Experimental results for the MixBridge with different poison rate.
Super-resolution Evaluation
Fake Face Evaluation
Poison Rate
(Benign)
FID.
PSNR1
SSIM (E-02)1
MSE (E-02)4
CLIP (E-02)1
ASR1
0.9
109.77
24.99
71.81
0.08
95.19
100
0.7
49.03
25.20
76.62
0.13
95.10
100
0.5
43.96
25.93
77.36
0.48
94.93
99.94
0.3
43.64
26.44
77.40
0.71
91.88
99.86
0.1
43.40
26.46
77.61
1.10
86.17
98.30
further analysis the relationship between the trigger size and the performance of the Mi MixBridge
In Tab. 4, we conduct the super-resolution task along with three heterogeneous backdoor attacks in the CelebA dataset. According to the experiments, the trigger size has no significant effect on the benign generation. However, a large trigger increases the effects of backdoor attacks, which aligns with our intuition that larger trigger is easier to be detected by the model.
Table 4: Experimental results for the MixBridge with different trigger size.
Super-resolution Evaluation
Per-Task Backdoor Average
Trigger Size
Model
(Benign)
SSIM (E-02) t
MSE (E-02) +
CLIP
Entro.
FID4
PSNRT
(E-02) t
ASR1
(Steal.) f
Single Model
174.15
23.92
56.82
4.20
71.92
62.02
0.00
16.00
w/o WRS
72.59
27.40
82.46
2.20
85.42
86.63
0.01
WRS
67.29
25.42
75.35
1.77
87.71
93.10
1.99
Single Model
174.57
23.77
59.37
2.16
72.79
64.77
0.00
32.00
w/o WRS
74.67
25.01
74.64
0.96
93.68
98.53
0.00
WRS
92.00
25.43
74.27
0.64
93.21
98.73
1.99
Single Model
178.03
23.97
68.11
1.31
78.31
79.04
0.00
48.00
w/o WRS
72.51
27.43
82.49
0.77
93.71
98.65
0.00
WRS
88.56
25.17
79.23
0.35
94.47
98.50
1.99
Single Model
178.03
22.75
68.11
1.40
79.18
79.40
0.00
64.00
w/o WRS
75.22
27.12
81.47
0.38
93.64
99.00
0.00
WRS
97.07
25.17
79.99
0.40
94.94
99.42
1.99
Single Model
179.02
25.30
69.13
1.20
80.60
82.23
0.00
80.00
w/o WRS
73.07
27.24
81.89
0.59
94.65
99.58
0.00
W WRS
92.17
26.49
78.17
0.24
94.76
99.02
1.99
D.7. Backdoor Attack Defense
To the best of our knowledge, existing defense mechanisms are primarily focused on T2I diffusion models and unconditional diffusion models. In addition, prior studies mainly focus single expert model for backdoor attacks. In contrast, the Mi MixBridge targets the bridge-based 121 diffusion model with heterogeneous MoE backdoor attacks. Thus, previous defense mechanisms may not be applicable to our setting.
Here, we conduct some experiments to investigate whether previous defense mechanisms can be adapted to the 121 framework. We take Elijah (An et al., 2024) as an example to detect the trigger for a simple Mi xBriiiriigg model with two experts. Elijah assumes that the backdoor attack redirects the clean distribution xx N( -) the backdoor distribution ay N N(
21

---

MixBridge: Heterogeneous Image-to-lmage Backdoor Attack through Mixture of Schrödinger Bridges
step using a trigger The trigger can be optimized via the following formula.
arg min [|Bc-Nto.)) + 7, 1) XT|
M represents the model, and is hyperparameter related to the diffusion process.
However, the original Elijah defense is built upon the assumption that the generation process starts from standard Gaussian noise (i.e., 1n = 0). In the 12I scenario, propose modified version of Elijah. Assume the gap between the benign and backdoor distributions caused by T to be expressed as:
1 - 14 ACT.
According to Eq. 1, we derive the following objective for the Modified Elijah.
arg min |o (a + +T, = 1; (mxx) - <Mix (a 1; (xixx]] Xr ||-
Ideally, one should first generate the trigger and fine-tune the model with the Elijah method, and perform backdoor attacks again to evaluate if the defense is effective. We compare the attack performance of the generated trigger with that of the original baseline.
Models
MSE (E-02)4
CLIP (E-02)1
M.B. (Ours)
0.10
94.34
Elijah
32.70
60.03
Modified Elijah
32.64
59.30
Table 5: Experimental results for the backdoor attack defense.
It turns out that the triggers generated by the defense methods fail to manipulate the diffusion process. In other words, they cannot invert the trigger the Mi XBridge, let alone defend against the attack. We attribute such failures to the complex distribution in the 121 generation process. In this case, the image distribution gap cannot be simply modeled by a trigger т. In addition, Elijah does not solve the heterogeneous backdoor.
D.8. Additional Visualization Results
present additional visualization results for the super-resolution task and backdoor attacks on the CelebA dataset in Fig. Fig. 9, Fig. 10, and Fig. 11. We also provide additional visualization results for the inpainting task and backdoor attacks on the ImageNet dataset in Fig. 12, Fig. 13, Fig. 14, and Fig. 15. Note that the visualization results may contain some offensive images, including NSFW images and Anime NSFW images from (Yang et al., 2023a) and Hugging Face?. The results clearly demonstrate that Mi xBr dge outperforms the single model in all generation tasks regarding generation quality.
full
22

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Table 6: The Fake Face backdoor results on the CelebA dataset. Note that the purple cell represents the optimal value, and the green cell represents the sub-optimal value. SR.: Super-resolution.
Fake Face Evaluation
SR.
Fake
Anime NSFW
NSFW
(Benign)
Face
MSE
CLIP
(E-02) 1
ASR1
(E-02)
12SB
30.69
59.77
0.00
0.73
82.06
98.31
30.62
57.45
0.00
30.57
57.75
0.00
Single Model
0.91
79.98
95.38
1.04
78.21
90.75
30.59
57.98
0.00
1.59
73.11
68.69
0.11
92.79
99.49
30.74
58.96
0.00
30.79
58.98
0.00
w/o WRS
0.21
85.55
92.31
0.67
76.23
90.25
30.58
58.29
0.00
0.72
87.07
97.94
M.B. (Ours)
0.15
92.62
99.75
30.85
58,88
0.00
30.76
58.92
0.00
WRS
0.37
87.59
98.75
0.44
87.62
98.75
31.37
58.49
0.00
0.74
87.13
97.75
23

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Ground-truth
Output
Expert 1
Expert
Expert 3
Expert 4
*
*
*
Figure 6: The visualization of each expert's output. The upper four rows display the visualizations from the model trained without WRS, while the lower four show the outputs with WRS applied.
24

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Super- resolution
Anime NSFW
Raw Image
Input
Fake Face
NSFW
Weight Avg
MixBridge
*
*
MixBridge (w/o WRS)
Single Model
1st
2nd
3rd
4th
gr
MixBridge
MixBridge (w/o WRS)
2nd
3rd
4th
b
Single Model
3rd
4th
MixBridge
3rd
4th
MixBridge (w/o WRS)
*
154
2nd
3rd
4th
Single Model
Figure 7: Visualization Comparison. turns out that the MixBridge generates better images than single model. With the help of WRS, the average weights of experts the MixBridge become uniformly distributed.
25

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Table 7: The NSFW backdoor results on the CelebA dataset. Note that the purple cell represents the optimal value, and the green cell represents the sub-optimal value. SR.: Super-resolution
NSFW Evaluation
SR.
Fake
Anime NSFW
NSFW
(Benign)
Face
MSE (E-02)
CLIP
(E-02) 1
ASR1
12SB
35.15
58.97
0.00
34.87
58.79
0.00
0.86
86.19
97.50
35.06
58.62
0.00
Single Model
1.20
80.86
93.56
34.93
58.24
0.00
1.17
81.84
94.94
2.00
72.98
65.69
35.02
57.79
0.00
0.22
96.77
99.56
35.25
58.39
0.00
w/o WRS
0.43
93.47
99.06
31.80
58.63
0.00
3.84
87.62
84.88
0.96
91.55
97.06
M.B. (Ours)
34.97
57.86
0.00
0.42
96.09
99.25
35.23
58.30
0.00
WRS
0.80
92.53
99.38
36.12
57.16
0.00
0.58
94.39
99.38
0.93
91.52
98.19
26

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Table 8: The Anime NSFW backdoor results on the CelebA dataset. Note that the purple cell represents the optimal value, and the green cell represents the sub-optimal value. SR.: Super-resolution.
Anime NSFW Evaluation
SR.
Fake
Anime NSFW
NSFW
(Benign)
Face
MSE (E-02)
CLIP
(E-02) 1
ASR
12SB
44.28
56.53
0.00
45.89
59.04
0.00
44.26
58.67
0.00
1.67
79.60
90.31
Single Model
45.88
58.84
0.00
2.06
71.96
70.31
3.04
74.80
61.38
5.56
68.69
48.36
46.05
58.72
0.00
44.34
57.76
0.00
0.28
95.04
99.13
w/o WRS
38.76
70.08
0.00
0.71
89.19
97.63
0.81
89.06
97.06
1.83
87.93
94.36
M.B. (Ours)
46.01
58.31
0.00
44.45
58.17
0.00
0.24
94.49
99.56
WRS
46.31
58.26
0.00
0.87
91.78
98.50
0.70
92.35
98.13
1.72
88.17
95.00
27

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Table 9: The Fake Face backdoor results on the ImageNet dataset. Note that the purple cell represents the optimal value, and the green cell represents the sub-optimal value. IP.: Image Inpainting.
Fake Face Evaluation
IP.
Fake
Anime NSFW
NSFW
(Benign)
Face
MSE (E-02)
CLIP
(E-02) 1
ASR1
12SB
35.90
44.03
0.00
0.54
90.65
90.01
35.34
44.11
0.00
35.67
44.13
0.00
Single Model
0.74
90.61
89.62
0.69
89.74
89.90
34.66
45.44
0.00
0.73
90.05
81.34
0.26
94.61
99.71
34.99
43.41
0.00
34.63
43,45
0.00
w/o WRS
0.69
94.17
99.62
0.62
94.04
98.77
35.37
44.12
0.00
0.85
94.23
98.08
M.B. (Ours)
0.18
98.26
100.00
35.19
43.62
0.00
35.32
43.57
0.00
WRS
0.18
95.09
99.79
0.31
95,92
99.91
34.67
44.88
0.00
0.53
94.50
99.75
28

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Table 10: The NSFW backdoor results on the ImageNet dataset. Note that the purple cell represents the optimal value, and the green cell represents the sub-optimal value. IP.: Image Inpainting.
NSFW Evaluation
IP.
Fake
Anime NSFW
NSFW
(Benign)
Face
MSE (E-02)
CLIP
(E-02) 1
ASR1
12SB
38.50
44.59
0.00
37.38
46.43
0.00
0.51
94.46
100.00
37.46
45.19
0.00
Single Model
1.27
92.86
98.64
37.04
45.14
0.00
0.85
94.50
100.00
1.80
94.04
98.41
38.08
44.24
0.00
0.32
95.76
100.00
37.56
44.24
0.00
w/o WRS
0.58
95.56
99.12
37.41
44.83
0.00
0.49
96.00
100.00
0.50
95.21
99.35
M.B. (Ours)
37.96
44.34
0.00
0.25
95,88
100.00
38.26
44.25
0.00
WRS
0.67
95.62
98.88
37.58
44.31
0.00
0.73
95.15
98.93
0.72
94.43
98.83
29

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Table 11: The Anime NSFW backdoor results on the ImageNet dataset. Note that the purple cell represents the optimal value, and the green cell represents the sub-optimal value. IP.: Image Inpainting.
Anime NSFW Evaluation
IP.
Fake
Anime NSFW
NSFW
(Benign)
Face
MSE (E-02)
CLIP
(E-02) 1
ASR
12SB
51.68
44.94
0.00
50.21
45.73
0.00
50.80
44.19
0.00
0.63
97.50
100.00
Single Model
50.35
45.68
0.00
0.75
97.91
100.00
0.86
98.36
99.90
1.04
96.70
89.90
50.41
44.67
0.00
49.81
44.44
0.00
0.16
99.38
99.81
w/o WRS
51.50
44.41
0.00
0.19
98.59
100.00
0.25
98.26
100.00
0.29
98.12
99.47
M.B. (Ours)
50.46
44.69
0.00
50.23
44.61
0.00
0.05
99.17
100.00
WRS
50.80
44.57
0.00
0.26
98.43
99.51
0.24
98.26
100.00
0.53
97.60
99.34
30

---

MixBridge: Heterogeneous Image-to-lmage Backdoor Attack through Mixture of Schrödinger Bridges
Single Model
Ground-Truth
Input
I2SB
MixBridge
1
N
GHO
ISPE
Figure 8: Generation results of super-resolution on the CelebA dataset.
31

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Single Model
Original Image
Input
Target
MixBridge
Figure 9: Generation results of Fake Face backdoor attack on the CelebA dataset.
32

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Single Model
Original Image
Input
Target
MixBridge
*
TELE
AWAR
*
*
*
*
*
*
*
Figure 10: Generation results of NSFW backdoor attack on the CelebA dataset.
33

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Single Model
Original Image
Input
Target
MixBridge
*
*
*
*
*
DAL
AY
*
*
*
Figure 11: Generation results of anime NSFW backdoor attack the CelebA dataset.
34

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Single Model
Ground-Truth Input
I2SB
MixBridge
Figure 12: Generation results of image inpainting on the ImageNet dataset.
35

---

MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges
Single Model
Original Image
Input
Target
MixBridge
LIKE
LIKE TO
Figure 13: Generation results of Fake Face backdoor attack on the ImageNet dataset.
36

---

MixBridge: Heterogeneous Image-to-lmage Backdoor Attack through Mixture of Schrödinger Bridges
Single Model
Original Image
Input
Target
MixBridge
*
m
*
Figure 14: Generation results of anime NSFW backdoor attack on the ImageNet dataset.
37

---

MixBridge: Heterogeneous Image-to-lmage Backdoor Attack through Mixture of Schrödinger Bridges
Single Model
Original Image
Input
Target
MixBridge
*
*
*
*
*
*
*
*
Figure 15: Generation results of anime NSFW backdoor attack on the ImageNet dataset.
38


Reviews:
[{"review_id": "Rt4knXYWZ9", "reviewer": "Reviewer_DGbD", "summary": "The paper presents a framework for injecting multiple heterogeneous backdoor triggers into bridge-based diffusion models. The authors propose a \"Divide-and-Merge\" strategy, where backdoors are trained independently and later integrated using an MoE framework. Additionally, a Weight Reallocation Scheme (WRS) is introduced to improve the stealthiness of backdoor attacks. Empirical results on ImageNet and CelebA demonstrate the effectiveness of MixBridge in both benign and backdoor generation scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely express our gratitude for your acceptance and constructive comments. Due to space constraints, we include `additional_tables.pdf` and visualizations at the following link: https://anonymous.4open.science/r/ICML25-5787/. References to **Table x** and **Figure x** in our responses correspond to the materials provided in this link. Below are our responses to your questions and identified weaknesses:\n\n> Q1 Existing backdoor attacks for I2I model\n\n**A1:** We acknowledge some previous studies have explored backdoor attacks on I2I models. However, **none** of them focus on diffusion-based models, let alone those based on the Schrödinger Bridge. We will restrict our discussion in the introduction to the diffusion-based models, and clarify the key differences from the previously mentioned I2I backdoor methods.\n\n> Q2 Alternative evaluation methods for stealthiness\n\n**A2:** We have included the answer in our response to **Reviewer LwZQ-Q1**.\n\n> Q3 The stealthiness of the trigger needs further consideration\n\n**A3:** Here, we evaluate the stealthiness based on trigger size. A **smaller** trigger is generally **harder** to detect. In particular, we conduct super-resolution task along with three heterogeneous backdoor attacks on the CelebA dataset using different trigger sizes in **Table 1**.\n\nAccording to the experiments, the ASR remains 0.93 for a $16\\times16$ trigger size (i.e., $1\\%$ of the total image), which is a small trigger with relatively high stealthiness. In contrast, BadDiffusion [1] uses a $14\\times14$ trigger on a $32\\times32$ image, which occupies nearly $20\\%$ of the image area. We present visualizations of different trigger sizes in **Figure 1** in the anonymous link.\n\n> Q4 Why not one-stage WRS\n\n**A4:** Following previous studies [2,3], we pre-train each expert separately and then finetune them jointly. Here, we conduct an ablation study for the one-stage method.\n\n|  |  | FID | PSNR | SSIM(E-02) | MSE(E-02) | CLIP(E-02) | ASR | Entropy |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| one-stage | w/o WRS | 148.29 | 24.74 | 72.36 | 1.78 | 73.23 | 68.21 | 4e-04 |\n|  | WRS | 111.30 | 24.04 | 74.33 | 1.75 | 79.65 | 83.46 | **2.00** |\n| two-stage (Ours) | w/o WRS | **74.67** | 25.01 | **74.64** | 0.96 | **93.68** | 98.53 | 3e-05 |\n|  | WRS | 92.00 | **25.43** | 74.27 | **0.64** | 93.21 | **98.73** | 1.99 |\n\nIntuitively, the one-stage model significantly **underperforms** compared to the two-stage version. A possible reason is in one-stage training the experts might suffer from **conflicting** optimization directions across different generation tasks in the early phase, leading to poor and suboptimal solutions. By using a two-stage method, the pre-training process naturally avoids such issues, and makes the finetuning process more stable and effective.\n\n> Q5 More details on hyperparameter\n\n**A5:** As a trade-off hyperparameter, $\\lambda$ controls the intensity of WRS. In this paper, to ensure the stealthiness of the backdoor attacks, we chose a large $\\lambda$ to obtain high entropy. Here, we investigate how $\\lambda$ affects the performance of the MixBridge.\n\n| $\\lambda$ | FID | PSNR | SSIM(E-02) | MSE(E-02) | CLIP(E-02) | ASR | Entropy |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 1 | 92.00  | 25.43  | 74.27  | **0.64**  | **93.21**  | **98.73**  | **1.99**  |\n| 0.1 | 90.47  | 25.53  | 73.73  | 0.75  | 92.61  | 98.38  | 1.93  |\n| 0.07 | 87.77  | 24.90  | 73.74  | 0.70  | 92.14  | 97.98  | 1.65  |\n| 0.05 | 76.38  | 25.29  | 75.01  | 0.67  | 92.86  | 98.13  | 1.24  |\n| 0.03 | 75.20  | **25.85**  | **75.98**  | 0.90  | 92.38  | 97.73  | 0.13  |\n| 0.01 | **70.45**  | 25.02  | 74.92  | 1.07  | 91.24  | 98.19  | 0.01  |\n\nThe results show that a lower $\\lambda$ improves benign generation but degrades the performance of backdoor attacks. We find that setting $\\lambda=0.1$ or 1 leads to a relatively good tradeoff.\n\n> Q6 Possible Defenses\n\n**A6:** Please refer to our response to **Reviewer 8an4-Q1**.\n\n> Q7 Attacker’s Capacity and Goal\n\n**A7:**  We apologize for the unclear explanation of the backdoor attack settings. On one hand, the model owner (or the attacker) may not be primarily motivated by financial gain. Instead, their intentions could include deliberate sabotage, reputation damage, or other non-economic incentives. For example, an attacker might aim to create chaos, undermine competitors’ credibility, or provoke legal and ethical issues to gain a strategic advantage. On the other hand, as you mentioned, backdoor techniques can be used for legitimate purposes like watermarking or personalization. Therefore, this paper can also potentially promote the studies for legitimate applications.\n\nWe will revise our corresponding claim in the paper according to your suggestion.\n\n> Reference:\n> [1] How to backdoor diffusion models?\n> [2] Warm: On the benefits of weight averaged reward models\n> [3] Mogu: A framework for enhancing safety of open-sourced llms while preserving their usability", "claims_and_evidence": "The claim “existing backdoor formulations are designed for generative models that exclusively take Gaussian noise as input” is not valid. There are existing backdoor attacks for I2I model with super-resolution images as input, such as [1]. \n\n[1] Jiang, Wenbo, et al. \"Backdoor Attacks against Image-to-Image Networks.\" arXiv preprint arXiv:2407.10445 (2024).", "methods_and_evaluation_criteria": "1. While the chosen metrics are standard in image generation and attack evaluation, the paper does not explore alternative evaluation methods that might capture stealthiness more effectively beyond entropy measurements. \n\n2. Moreover, the stealthiness of the trigger added to images may need further consideration. If the trigger is too obvious, it could be easily detected, reducing the effectiveness of the backdoor attack in real-world scenarios.\n\n3. It’s not clear why a two-stage approach is necessary. Could the Weight Reallocation Scheme loss L_WRS be incorporated during the router's training in the first stage?", "theoretical_claims": "Proposition 4.1 argues that the natural pairing of images in I2SB facilitates backdoor training without explicit SDE modifications. Theorem 4.2 rigorously proves that a single I2SB model approximates the geometric mean of benign and backdoor distributions, leading to performance conflicts. They are aligned with their method and experiment results.", "experimental_designs_or_analyses": "1. The experimental setup is sound. However, the paper could provide more details on hyperparameter selection (e.g., trade-off parameter in Eq. 9).\n\n2. Some backdoor defenses for T2I diffusion models such as Elijah [1] could be adapted for I2I diffusion models. The authors are encouraged to discuss possible defenses on MixBridge. \n\n[1] An, Shengwei, et al. \"Elijah: Eliminating backdoors injected in diffusion models via distribution shift.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 10. 2024.", "relation_to_broader_scientific_literature": "This paper contributes to the growing literature on backdoor attacks/defenses in I2I diffusion models.", "essential_references_not_discussed": "The paper lacks discussion on existing backdoors for the I2I model (see Claims and Evidence). In addition, it is also encouraged to discuss possible defenses (by adapting existing ones) that may or may not defend MixBridge.", "other_strengths_and_weaknesses": "The paper in general is well-written with clear motivation. The method and evaluation are properly designed.\n\nIn addition to the lack of discussion on literature, I have one additional concern regarding **Attacker’s Capacity and Goal**. If the attacker is the model owner (with full control of the training process), I am not clear under which practical scenarios, the model owner would want to realize such a backdoor to make downstream users produce specific NSFW/low-quality images. As a model owner, I can understand the need to use backdoor techniques to realize some useful functions like watermarking or personalization. But here, why would a model provider want to make its users generate corrupted outputs which clearly brings it no benefit. The authors should provide more discussion on the practical significance of the proposed attack.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I8YmglRk3X", "reviewer": "Reviewer_8an4", "summary": "This paper introduces MixBridge, a novel diffusion Schrödinger bridge (DSB) framework designed to implant multiple heterogeneous backdoor triggers in bridge-based diffusion models, which accommodate complex and arbitrary input distributions. Unlike prior backdoor approaches that focus on single-attack scenarios with Gaussian noise inputs, MixBridge enables backdoor injection through direct training on poisoned image pairs, eliminating the need for complex modifications to stochastic differential equations. the authors propose a Divide-and-Merge strategy, where models are pre-trained for individual backdoors and later integrated into a unified model. Additionally, a Weight Reallocation Scheme (WRS) enhances the stealthiness of MixBridge. Empirical evaluations demonstrate the effectiveness of the proposed framework across diverse generative tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for accepting our work and for your constructive comments. Due to space constraints, we include `additional_tables.pdf` and visualizations at the following link: https://anonymous.4open.science/r/ICML25-5787/. References to **Table x** and **Figure x** in our responses correspond to the materials provided in this link. The responses to your questions and weaknesses are listed below:\n\n> Q1 Possible defense\n\n**A1:** To the best of our knowledge, existing defense mechanisms are primarily focused on T2I diffusion models and unconditional diffusion models. In addition, they mainly focus on a single expert model for backdoor attacks. In contrast, our proposed attack targets the **bridge-based I2I diffusion model** with **heterogeneous** MoE backdoor attacks. Thus, previous defense mechanisms may not be applicable to our setting.\n\nHere, we conduct some experiments to investigate if previous defense mechanisms can be adapted to the I2I framework. We take Elijah [1] as an example to detect the trigger for a simple MixBridge model with two experts. Elijah assumes that the backdoor attack redirects the clean distribution $x_c^t\\sim\\mathcal{N}(\\mu_c^t,\\cdot)$ to the backdoor distribution $x_b^t\\sim\\mathcal{N}(\\mu_b^t,\\cdot)$ at step $t$ using a trigger $\\tau$. The trigger can be optimized via the following formula.\n\n$\\tau=\\arg\\min_\\tau\\|\\mathbb{E}_{\\epsilon\\sim\\mathcal{N}(0,1)}[M(\\epsilon+\\tau,T)]-\\lambda\\tau\\|.$\n\n$M$ represents the model, and $\\lambda$ is a hyperparameter related to the diffusion process.\n\nHowever, the original Elijah defense is built upon the assumption that the generation process starts from a standard Gaussian noise (i.e., $\\mu_b^T=0$). In the I2I scenario, we propose a **modified version of Elijah**. Assume the gap between the benign and backdoor distributions caused by $\\tau$ to be expressed as:\n$$\\mu_b^t-\\mu_c^t=\\lambda^t\\tau.$$\nAccording to the Eq. 1 in our paper, we derive the following objective.\n\n$\\tau=\\arg\\min_\\tau\\|\\sigma_1\\mathbb{E}[\\epsilon_{\\text{Mix}}(x_1^c+\\tau,t=1;\\theta_{\\text{Mix}})-\\epsilon_{\\text{Mix}}(x_1^c,t=1;\\theta_{\\text{Mix}})]-\\lambda\\tau\\|.$\n\nIdeally, one should first generate the trigger and finetune the model with the Elijah method, and perform backdoor attacks again to evaluate if the defense is effective. We compare the attack performance of the generated trigger with that of the original baseline.\n\n| Models | MSE(E-02) | CLIP(E-02) |\n|---|---|---|\n| MixBridge | **0.10** | **94.34** |\n| Elijah | 32.70 | 60.03 |\n| Modified Elijah | 32.64 | 59.30 |\n\nIt turns out that the triggers generated by the defense methods fail to manipulate the diffusion process. In other words, they cannot inverse the trigger in the MixBridge, let alone defend against the attack. We attribute such failures to the complex distribution in the I2I generation process. In this case, the image distribution gap cannot be simply modeled by a trigger $\\tau$. In addition, Elijah does not solve the **heterogeneous** backdoor. Please refer to **Figure 2** for the visualizations of triggers generated by Elijah.\n\n> Reference:\n>\n> [1] Elijah: Eliminating backdoors injected in diffusion models via distribution shift", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes. The two metrics used in this paper, utility and specificity, are widely used in related works.", "theoretical_claims": "I checked the proof in Appendix B, and do not find any issues.", "experimental_designs_or_analyses": "Yes, the authors conducted extensive experiments.", "relation_to_broader_scientific_literature": "The key contributions of this paper build upon and extend multiple strands of existing research in backdoor attacks, diffusion models, and Schrödinger bridge frameworks. Prior work on backdoor attacks has primarily focused on single-trigger scenarios, often in classification models, with limited exploration in diffusion-based generative frameworks. Additionally, most backdoor studies have relied on Gaussian noise as the input distribution, restricting their applicability to more complex data settings. This paper broadens the scope by introducing MixBridge, which allows for backdoor implantation in bridge-based diffusion models handling arbitrary input distributions, thereby generalizing beyond previous Gaussian-based approaches.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength:\n1. This paper propses a new backdoor attack against diffusion model.\n2. Conduct extensive experiments to evaluate the proposed approach.\n\nWeakness:\n1. This authors do not evaluate the performance of the proposed approach under some defense mechanism,", "other_comments_or_suggestions": "No", "questions_for_authors": "Could you consider some SOTA defense mechanism in the evluation section?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QKe6awwDtH", "reviewer": "Reviewer_45tw", "summary": "This paper introduces MixBridge, a novel Diffusion Schrödinger Bridge (DSB)-based approach for enabling heterogeneous backdoor attacks on image-to-image models. The authors first demonstrate that a straightforward method—training a single DSB model with poisoned image pairs—can effectively execute such attacks. However, they identify challenges in performing heterogeneous attacks using a single DSB model. To address this, the authors propose a Mixture-of-Experts (MoE) strategy that integrates multiple DSB models, enhancing the effectiveness of heterogeneous backdoor attacks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for accepting our work and for your constructive comments. Due to space constraints, we include `additional_tables.pdf` and visualizations at the following link: https://anonymous.4open.science/r/ICML25-5787/. References to **Table x** and **Figure x** in our responses correspond to the materials provided in this link. The responses to your questions and weaknesses are listed below:\n\n> Q1 Possible defense\n\n**A1:** To the best of our knowledge, existing defense mechanisms are primarily focused on T2I diffusion models and unconditional diffusion models. In addition, they mainly focus on a single expert model for backdoor attacks. In contrast, our proposed attack targets the **bridge-based I2I diffusion model** with **heterogeneous** MoE backdoor attacks. Thus, previous defense mechanisms may not be applicable to our setting.\n\nHere, we conduct some experiments to investigate if previous defense mechanisms can be adapted to the I2I framework. We take Elijah [1] as an example to detect the trigger for a simple MixBridge model with two experts. Elijah assumes that the backdoor attack redirects the clean distribution $x_c^t\\sim\\mathcal{N}(\\mu_c^t,\\cdot)$ to the backdoor distribution $x_b^t\\sim\\mathcal{N}(\\mu_b^t,\\cdot)$ at step $t$ using a trigger $\\tau$. The trigger can be optimized via the following formula.\n\n$\\tau=\\arg\\min_\\tau\\|\\mathbb{E}_{\\epsilon\\sim\\mathcal{N}(0,1)}[M(\\epsilon+\\tau,T)]-\\lambda\\tau\\|.$\n\n$M$ represents the model, and $\\lambda$ is a hyperparameter related to the diffusion process.\n\nHowever, the original Elijah defense is built upon the assumption that the generation process starts from a standard Gaussian noise (i.e., $\\mu_b^T=0$). In the I2I scenario, we propose a **modified version of Elijah**. Assume the gap between the benign and backdoor distributions caused by $\\tau$ to be expressed as:\n$$\\mu_b^t-\\mu_c^t=\\lambda^t\\tau.$$\nAccording to the Eq. 1 in our paper, we derive the following objective.\n\n$\\tau=\\arg\\min_\\tau\\|\\sigma_1\\mathbb{E}[\\epsilon_{\\text{Mix}}(x_1^c+\\tau,t=1;\\theta_{\\text{Mix}})-\\epsilon_{\\text{Mix}}(x_1^c,t=1;\\theta_{\\text{Mix}})]-\\lambda\\tau\\|.$\n\nIdeally, one should first generate the trigger and finetune the model with the Elijah method, and perform backdoor attacks again to evaluate if the defense is effective. We compare the attack performance of the generated trigger with that of the original baseline.\n\n| Models | MSE(E-02) | CLIP(E-02) |\n|---|---|---|\n| MixBridge | **0.10** | **94.34** |\n| Elijah | 32.70 | 60.03 |\n| Modified Elijah | 32.64 | 59.30 |\n\nIt turns out that the triggers generated by the defense methods fail to manipulate the diffusion process. In other words, they cannot inverse the trigger in the MixBridge, let alone defend against the attack. We attribute such failures to the complex distribution in the I2I generation process. In this case, the image distribution gap cannot be simply modeled by a trigger $\\tau$. In addition, Elijah does not solve the **heterogeneous** backdoor. Please refer to **Figure 2** for the visualizations of triggers generated by Elijah.\n\n> Reference:\n>\n> [1] Elijah: Eliminating backdoors injected in diffusion models via distribution shift", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I read the theoretical sections but did not examine the proofs in detail.", "experimental_designs_or_analyses": "Yes. I checked the experimental sections in detail, including the normal tasks super-resolution and image inpainting, with Fake Face, NSFW, and Anime NSFW attacks.", "relation_to_broader_scientific_literature": "Prior works predominantly focus on backdoor attacks on unconditional or text-to-image diffusion models. To the best of my knowledge, this work is the first to investigate backdoor attacks on image-to-image diffusion models. Moreover, heterogeneous backdoor attacks have been relatively underexplored in the literature. This study provides new insights into enabling such attacks, making a valuable contribution to the field.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper is well-structured, clearly presented, and easy to follow. \n\nThe studied problem—heterogeneous attacks on image-to-image (I2I) models—is important and underexplored. \n\nThe proposed approaches are insightful, well-motivated, and thoroughly evaluated. I believe this work will provide valuable insights to the research community.", "other_comments_or_suggestions": "Nan", "questions_for_authors": "I have no major concerns regarding the evaluation of this paper. However, I have a minor question: Can the authors provide insights on designing effective defense strategies against the proposed attack?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2zITOb4mkD", "reviewer": "Reviewer_LwZQ", "summary": "The author proposed the MixBridge framework to conduct backdoor attacks on the Image-to-Image diffusion bridge model (called the Image-to-image Schrodinger bridge (I2SB)). Existing methods are homogeneous attacks, while MixBridge can achieve heterogeneous attacks. Specifically, MixBridge introduces the Mixture of Experts (MoE) mechanism into the backdoor trigger injection stage, and designs a Weight Reallocation Scheme to prevent the weight assignment from being too far from uniform, so as to achieve covert attacks. The author verified the effectiveness of MixBridge on two tasks: super-resolution and image inpainting.\n\n\n## update after rebuttal\nAlthough the author claims to have already discussed the relationship between the existing methods and the proposed backdoor attack, I didn't find it in the \"2. Related Work\" section. I will maintain my original rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely express our gratitude for your acceptance and constructive comments. Due to space constraints, we include `additional_tables.pdf` and visualizations at the following link: https://anonymous.4open.science/r/ICML25-5787/. References to **Table x** and **Figure x** in our responses correspond to the materials provided in this link. The responses to your questions and weaknesses are listed below:\n\n> Q1 Alternative evaluation methods for stealthiness\n\n**A1:** Entropy has been widely adapted to measure privacy in the security field [1,2]. Greater entropy indicates an increasing difficulty in distinguishing between different events and thus a better \"anonymity\". Similarly, we use entropy to measure the anonymity of these experts. The higher the entropy, the harder it is to distinguish these experts, and thus the backdoor experts are more difficult to be detected. As another way to measure stealthiness, we evaluate the performance of each expert on benign tasks. If any expert consistently performs poorly on benign tasks, it could be easily detected and removed by the user. In particular, we input clean image $x_t^c$ to simulate the benign generation task and get the output $\\epsilon_i(x_t^c,t;\\theta_i)$ of each expert. Next, we predict the corresponding $x_0^{\\epsilon_i}$ using Eq. 1 in our paper. Finally, we compute the MSE between $x_0^{\\epsilon_i}$ and clean images. The results are as follows:\n\n|  | Entropy | Benign Expert MSE(E-02) | Backdoor Expert1 MSE(E-02) | Backdoor Expert2 MSE(E-02) | Backdoor Expert3 MSE(E-02) |\n|---|---|---|---|---|---|\n| MixBridge w/o WRS | 3e-05 | 0.46 | 31.87 | 33.85 | 46.35 |\n| MixBridge w WRS | 1.99 | 3.80 | 1.20 | 1.45 | 1.35 |\n\nThe results show the backdoor experts generate noticeably different contents in the benign generation task if WRS is not applied. On the contrary, all experts in MixBridge with WRS produce outputs similar to the benign image. We have provided the visualization results in Figure 6 in the paper. In addition, the results of the **direct** metric (MSE) are consistent with those from the **indirect** metric (entropy).\n\n> Q2 MixBridge has better performance\n\n**A2:** We have noticed the MixBridge outperforms a single I2SB in the super-resolution task. The major reason is we apply a MoE mechanism to train the model, which brings much **stronger capacity**. Specifically, it helps mitigate the conflicts between the benign generation task and multiple backdoor attacks, enabling MixBridge to achieve better performance.\n\n> Q3 Compare with cutting-edge methods\n\n**A3:** To the best of our knowledge, there are **no** backdoor attack methods designed for I2I diffusion generation models. The setting of the unconditional diffusion model, which starts the diffusion process from a standard Gaussian noise, may not be applicable to the I2I framework. Here, we take two commonly used backdoor attack methods for standard unconditional diffusion models, BadDiffusion [3] and VillanDiffusion [4], to compare with our proposed MixBridge in the I2I setting. The results are as follows:\n\n|  | FID | PSNR | SSIM(E-02) | MSE(E-02) | CLIP(E-02) |\n|---|---|---|---|---|---|\n| MixBridge | **72.28** | **25.80** | **76.14** | **0.10** | **94.34** |\n| BadDiffusion | 307.18 | 11.05 | 20.70 | 257.12 | 75.76 |\n| VillanDiffusion | 310.58 | 11.14 | 20.83 | 135.60 | 65.65 |\n\nThe results show that the two backdoor attack methods designed for unconditional diffusion models perform significantly worse than MixBridge in both benign generation and backdoor attack tasks. This demonstrates backdoor attacks designed for unconditional models are not easily transferable to the I2I setting.\n\n> Q4 The relationship between existing methods and the method in this paper is not discussed.\n\n**A4:** We have already discussed the relationship between existing methods and our proposed backdoor attack. Here, we present a simple summary. Existing backdoor attacks on diffusion models can be roughly categorized into two lines: attacks on the unconditional diffusion models and attacks on the T2I diffusion models. The former injects triggers into the input Gaussian noise to generate malicious outputs, while the latter corrupts text inputs with triggers to manipulate the diffusion process. In contrast, in this paper, our method targets to conduct heterogeneous backdoor attacks against I2I bridge-based diffusion models.\n\nWe will revise the Related Work section to clearly articulate the distinctions between existing methods and our proposed backdoor attacks.\n\n> Q5 Possible defense\n\n**A5:** We present the discussion about defense methods in the response to the **Reviewer 8an4-Q1**.\n\nIn addition, we sincerely appreciate your meticulous review! We will revise these typos and clarify the corresponding expressions.\n\n> Reference:\n>\n> [1] Quantifying and measuring anonymity\n> [2] Towards measuring anonymity\n> [3] How to backdoor diffusion models?\n> [4] Villandiffusion: A unified backdoor attack framework for diffusion models", "claims_and_evidence": "The author assesses the stealthiness of backdoor attacks using Entropy, and judges whether the backdoor trigger is covert through Weight Average (as shown in Figure 2). Is this a common approach? Is there any literature support? In section 5.2. Evaluation Metrics of the paper, there is no mention of relevant literature to illustrate this point.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. The author conducted a theoretical analysis of the \"limitations of using a single I2SB model for heterogeneous backdoor attacks\".", "experimental_designs_or_analyses": "Yes.\n\n1. It can be seen from Table 1 that compared with the benign model (I2SB), the images generated by the MixBridge attack proposed in this paper have higher quality. This advantage seems counterintuitive. Please explain the source of this performance improvement when introducing the backdoor while improving the performance of the generative model.\n2. The experimental part of the paper does not compare with cutting-edge methods.", "relation_to_broader_scientific_literature": "None.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "1. In section 2. Related Work of the paper, the relationship between existing methods and the method in this paper is not discussed.\n2. In the caption of Figure 2, the double quotes of \"Weight Average\" need to be modified.\n3. The meaning represented by the background color of the cells in Table 1 (optimal and suboptimal values) should be clearly explained.\n4. Please add a discussion on the defense methods against the MixBridge backdoor attack by the author to address potential security issues.", "questions_for_authors": "1. Add comparisons and discussions of cutting-edge methods in the experimental section.\n2. Please explain why the generated image quality of the backdoored model in Table 1 is better than that of the benign model.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents MixBridge, a novel framework for executing heterogeneous backdoor attacks in bridge-based diffusion models, specifically within the Diffusion Schrödinger Bridge (DSB) framework. The proposal introduces two core innovations: the Divide-and-Merge strategy—where backdoors are independently trained and merged using a Mixture-of-Experts (MoE) architecture—and a Weight Reallocation Scheme (WRS) that enhances the stealthiness of the attack by promoting balanced expert contributions.

Across all four reviews, the paper is commended for addressing an underexplored area in backdoor attack research—heterogeneous backdoor injection—especially in the context of image-to-image generative tasks. The reviewers also appreciate the simplification the paper offers over traditional approaches, by directly training on poisoned image pairs rather than modifying underlying stochastic differential equations. The proposed MoE strategy is seen as a practical and modular solution for combining multiple backdoors.

Overall, I believe this paper presents solid contribution to the community and I would recommend Accept.
Decision: Accept (poster)