Secure Inference for Diffusion Models via Unconditional Scores
Authors: Jaeyun Song
Geondo Park
Uigyu Kim
Joonhyung Park
Eunho Yang
Venue: ICLR 2026 Poster (2026)
Keywords: privacy-preserving inference
diffusion models
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: As diffusion model-based services expand across various domains, safeguarding client data privacy has become increasingly critical. While fully homomorphic encryption and secure multi-party computation enable privacy-preserving inference, their high computational overhead poses challenges for large-scale diffusion applications. Recent work alleviates computational costs by substituting non-linear operations with low-degree polynomial approximations. While such relaxations reduce latency, they incur significant degradation in generative fidelity, and inference remains considerably slower than plaintext execution. To further accelerate secure inference while preserving performance, we explore more relaxed approximations and propose a score-correction framework that rectifies the conditional score shift induced by the relaxed approximation, rather than decreasing the approximation error itself. The key insight is that unconditional generation can be executed without approximation and thus provides a high-fidelity score signal. Leveraging this unconditional score as corrective guidance enables more relaxed approximations while preserving semantic and perceptual quality. In experiments, we demonstrate that our method significantly alleviates the performance degradation caused by relaxed approximations across various benchmarks.

Published as a conference paper at ICLR 2026
SECURE INFERENCE FOR DIFFUSION MODELS VIA UN- CONDITIONAL SCORES
Jaeyun Song', Geondo Park?, Uigyu Kim², Joonhyung Park2, Eunho Yang. KRAFTON', Korea Advanced Institute of Science and Technology (KAIST)², AITRICS jaeyun. song@krafton. com, eur.hoykkiist.. ac.kr
ABSTRACT
As diffusion model-based services expand across various domains, safeguarding client data privacy has become increasingly critical. While fully homomorphic encryption and secure multi-party computation enable privacy-preservin infer- ence, their high computational overhead poses challenges for large-scale diffusion applications. Recent work alleviates computational costs by substituting non-linear operations with low-degree polynomial approximations. While such relaxations duce latency, they incur significant degradation in generative fidelity, and inference remains considerably slower than plaintext execution. To further accelerate secure inference while preserving performance, we explore more relaxed approximations and propose score-correction framework that rectifies the conditional score shift induced by the relaxed approximation, rather than decreasing the approximation error itself. The key insight is that unconditional generation out approximation and thus provides a Sigh-fidelity score signal. Leveraging this unconditional score as corrective guidance enables more relaxed approximations while preserving semantic and perceptual quality. In experiments, we demonstrate that our method significantly alleviates the performance degradation caused by relaxed approximations across various benchmarks.
be executed with-
INTRODUCTION
Modern conditional diffusion models show impressive performance in generating images and are employed across a wide range of applications. Notably, Stable Diffusion (Rombach et al., 2022), one of the leading models, can produce high-quality images from user prompts or visual inputs such as sketches and key points. Building on the recent success of diffusion models, there is increasing interest in their application in privacy-sensitee In the medical field, they are utilized for translating MRI to CT, denoising medical images, and medical anomaly detections. Additionally, DALL-E (Ramesh et al., 2022) and Midjourney have become popular as commercial tools for content generation and creative industries.
fields, such as medical domain and content creation.
Despite these advancements, the substantial computational costs of image generation typically necessitate processing on powerful remote servers rather than local devices. This reliance on external servers introduces potential privacy concerns, as both user inputs and corresponding outputs may be exposed to the model provider. These concerns are especially critical in the medical domain, where inputs consist of sensitive information such as patient profiles and medical images, and outputs also include critical data such as CT scans in image translation tasks or enhanced images in denoising tasks. While sending data to a remote server raises privacy concerns for users, performing inference on client devices, on the other hand, exposes the model provider to the risk of model leakage. This dual privacy challenge complicates the deployment of diffusion models in privacy senssiivv fields.
Faced with the issue, CipherDM (Zhao et al., 2024) is the first to incorporate secure multi-party computation (MPC) (Yao, 1986) into diffusion models. Since the computation of non-linear operations a major efficiency bottleneck in MPC, CipherDM approximates the non-linear activations and exponential function with polynomials. Although this strategy achieves substantial latency reductions, causes significant performance degradation, and the overall latency remains significantly higher than that of plaintext execution, indicating that further acceleration is still required.

---

Fully homomorphic encryption (FHE) (Gentry, 2009) and secure multi-party computation (MPC) inference. FHE allows operations to be Published as a conference paper at ICLR 2026
To further decrease the latency while maintaining performance, investigate more aggressive approximations compared to the previous works, and propose a method that corrects approximation- induced score corruption rather than merely reducing individual approximation errors. The key insight is that unconditional generation can be executed without approximation, providing a high-fidelity score distribution that be used corrective guidance. By leveraging this unconditional score to steer the corrupted conditional score toward its true distribution, our framework mitigates score displacement and enables more aggressive polynomial approximations while preserving generation quality. Empirical results show that our approach significantly improves performance on binary MNIST with CipherDM, and on Flickråk (Hodosh et al., 2013) and MSCOCO (Lin et al., 2014 4) with Stable Diffusion (Rombach et al., 2022) under highly relaxed approximations, using MPC frameworks.
Our contribution is threefold:
We investigate relaxed polynomial approximations for non-linear activations and the ex- ponential function in diffusion models and observe operation-level latency reductions of 4436 a 2PC setting.
We propose novel score-gutee goredggi method that exploits the unconditional generation score to correct the approximation-coruug
conditional score.
Through extensive experiments, we show that the proposed method significantly mitigates performance degradation caused by relaxed approximations across multiple datasets.
PRELIMINARIES
2.1 DIFFUSION MODEL
Diffusion models (Sohl-Dickstein et al., 2015) learn data distribution Pdata by denoising variable following normal distribution. Diffusion models consist of two main processes: a forward process, where the data distribution is progressively transformed into a Gaussian distribution by incrementally adding noise, and a reverse process, where the Gaussian distribution is gradually transformed back into the data distribution by iteratively removing noise, guided by signals predicted by model. Therefore, the reverse process serves as the main mechanism for generating data. To learn the reverse process, the models are trained produce the variables from noise over T steps. Specifically, let Xy,..., X1 be variables where the initial noise, Xy, follows Gaussian distribution and X9 ~ Paau- For the model parameter 8, the previous variable is obtained by the reverse process as follows:
po (-xx-1|x)) ) == Af (Xt-15 ps (Xt, (), Ea (Xx, ())
In DDPM (Ho et al., 2020), the mean and covariance are predicted as:
Br
pe tx,,t)
VT at co(x);))
Eg (xx, BeI,
/ /ar
where B,,..., 3r are variance schedules, 04 = 1 3r, āi = II' ag and oe = / Br Since obtaining the mean via prediction (ea (x+: ()) in Equation 2 is more effective compared directly predicting the mean (po (xt, ()), models are trained to predict e.
However, diffusion models are computationally intensive, the diffusion process operates the pixel level, resulting in substantial memory and computational demands, particularly for high- resolution images. To reduce the computational cost, Latent Diffusion Model (LDM) (Rombach et al., 2022) performs diffusion processes in the latent space. This latent space has lower dimensionality and is constructed using an autoencoder. During inference, the initial noise Zy in the latent space progressively denoised following the reverse process to obtain which aligns with the latent space distribution induced by the data distribution and the autoencoder's encoder. Finally, 2o decoded back into the original data space as Xo using the autoencoder's decoder.
2.2 PRIVACY-PRESERVIIY INFERENCE
(Yao, 1986) widely adopted for
2

---

Published as a conference paper at ICLR 2026
10
900
2700
180
900
Precse Approx
Loose Approx
(GB)
+
500
1800
120
h20
Lazeniy
650
13500
90-
430
10:
Comm.
150
450
30
1.00
GeLU
Exp
SIDV
GetV
Exp
SIUU
GeLU
Exp
Mean Squared Error
(b) Latency
Volume
Figure 1: Comparison of mean squared error (MSE), latency (s), and communication volume (GB) for Stable Diffusion under two-party computation when reducing the polynomial degree the number of piecewise segments for Si GeLU, and the exponential function softmax. Gray bars show precise approximations, and green hars show the relaxed degree/segment variants.
performed directly on encrypted data, ensuring that the decrypted result nearly identical to the output of plaintext data, On the other hand, secure MPC allows multiple parties to jointly compute operations on combined data while preserving the privacy of each party's data. In secure MPC, secret data is divided into multiple shares and distributed among participants (Damgård et al., 2012), ensuring that reconstructing the original data is infeasible without all shares. Computations are executed these shares, ensuring privacy is maintained throughout the process. Despite these guarantees, both approaches incur significantly higher latency than plaintext execution due to heavy computational overhead in FHE and intensive communication in MPC.
These private inference methods are commonly used in machine learning-as-B8 as-viservic scenarios. As modern models now often exceed one billion parameters, latency has become critical concern in such scenarios. Specifically, the evaluation of non-linear operations is major efficiency bottleneck, motivating various approaches mitigate the associated cost. Polynomial approximations and it- erative algorithms have therefore been introduced to reduce computational overhead (Pang et al., 2024; Lu al., 2025). However, for diffusion models, these methods often result performance degradation, even on simple datasets (Zhao et al., 2024). The cumulative effect of small approxima- tion errors can mislead the model, causing it to produce noisy and low-quality images. Therefore, adopting more relaxed polynomial approximations, such as further lowering the polynomial degree, while maintaining generation quality, remains a critical challenge for secure inference diffusion models. To address this challenge, propose novel guidance strategy that leverages unconditional generation to correct the approximation-comm corruptedd conditional score.
SECURE INFERENCE FOR DIFFUSION MODEL WITH SCORE GUIDANCE
We now investigate recent approximation techniques, explore a relaxed variant, and analyze how reducing the polynomial degree the number of approximation segments affects both latency and performance in Section 3.1. Based on these observations, we introduce a method that corrects score shifts caused by approximation errors by leveraging the unconditional score in Section 3.2.
3.1 PERFORMANCE DEGRADATION UNDER RELAXED APPROXIMATIONS
First, we analyze the approximation accuracy and latency of non-linear operations in latent diffusion models, focusing on SiLUU, GeLU (Hendrycks Gimpel, 2016), and the exponential function. For evaluation, we report mean-squared error (MSE), total runtime, and communication volume for single forward step of the latent diffusion model (excluding the text encoder and VAE decoder) of Stable Diffusion (Rombach et al., 2022) executed under two-party computation (2PC) setting using CrypTen (Knott et al., 2021). We use Linux traffic control to simulate a local-ares area network with a one- way bandwidth of 3 Gbps and 0.3 ms round-trip latency an NVIDIA RTX A6000 GPU (48GB VRAM). For approximations, we adopt the polynomial schemes of BumbleBee (Lu et al., 2025) for SiLU, GeLU, and the exponential function, and those of CrypTen for other non-linear operations.
3

---

Published as a conference paper at ICLR 2026
Precise-Approx
AA
Loose-Approx
Loose-Approo + Ours
This wite metal rack
baby get with beau syes haaddnng next
Two sheep standing
This man playing
Two cows standing gaadyy
sandaals.
Figure 2: Comparison of text-to toniaee samples generated by a precisely approximated model (Precise- Approx), loosely approximated model (Lpose-Approx), and loosely approximated model with our method Loose-Appros Ours).
For SiLU, in particular, the input domain is partitioned into four segments and approximated by low-degree polynomials: the first segment [-8, -4) uses a quadratic (degree-2) polynomial, the second segment [-4, 4) employs a degree-6 polynomial, and the remaining regions are clipped to constant or linear forms. The piecewise approximation defined as
0,
if r < - if - 8 < -4
azz² + aix! + au.
Precise-SiLU(x)
beg byx b222 dy + bo. x
if 4 x <4
if 2 4,
where ao; 01,, (22 bo....y , denote coefficients optimized minimize MSE. The corresponding approximations for GeLU and the exponential function are detailed in Appendix A.
To further reduce latency, lowering the polynomial degree or the number of piecewise segments are commonly used. Specifically, we reduce the number of segments for SiLU and GeLU from four to three, lower the polynomial degree GeLU from six four, and apply a fourth-degree polynomial approximation to the exponential function in the second segment. The relaxed piecewise approximation for SiLU given by
0,
if -5.
Loose-SiLLU(a) =
cgx + c6x6 + cox + o2 + C1 + co.
- 5.4 x 4
xr,
if x 2 4,
where co. are the optimal coefficients minimizing mean squared errors. The corresponding relaxed approximations for GeLU and the exponential function are provided in Appendix A.
As shown in Figure 1, these relaxed approximations reduce both runtime and communication cost, achieving 14.05-35.604 latency reduction and 13.51-33.33% communication reduction. How- ever, this efficiency gain comes at the expense of accuracy: the approximation error increases by 2.55-26.255 26xxx, leading to significant fidelity degradation. As illustrated in Figure 2, naively applying relaxed approximations often produces blurred or structurally inconsistent images (for example, sheep with an excessive number of legs or misdrawn objects such as a stuffed bear when the prompt requests a baby girl). Additional qualitative comparisons are provided in Figure 5. One possible solution is to fine-tune the model after applying lower-degree approximations, but for large diffusion models, this approach is prohibitively expensive, requiring retraining models hundreds of millions of images. To address this challenge, we aim to correct the score corruption induced by relaxed approximations at inference time, rather than merely reducing the approximation errors themselves.
4

---

Published as a conference paper at ICLR 2026
Server
Plaintext Space
Latent Dif'usion Model
Regeat
Splto Suidice
Ciphertext Space
Eneryption
Latent Diffusion Model
Decoder
Enerypted
(Aporommetci)
Output
Noice
Repest
Unsondttional Samplng
Text Encoder ipprooematedi
Client
Encryption
Decryption
Toker
Input Text
Embedding
Output
Figure 3: The overview of our scenarios. The client sends the input to the server securely (via secret sharing or homomorphic encryption), which processes and returns the result. Denoising process occurs securely, with the non-linear functions the diffusion model approximated as linear. Next, our score guidance from unconditional scores is utilized to correct score shifts derived from approximation errors in conditional generation.
3.2 SCORE GUIDANCE FROM UNCONDITIONAL GENERATION
In the previous analysis, we observed that even small absolute approximation errors can accumulate across denoising steps, degrading image quality under more relaxed approximations. To mitigate this effect, we propose score guidance from unconditional generation, which refines the corrupted conditional scores by leveraging the score produced by unconditional generation. The key insight is that unconditional generation, executed entirely in the plaintext space, is free from approximation errors and can therefore provide a high-fidelity direction in the score space. The overall process is depicted Figure 3.
Specifically, our goal is to adjust the conditional score to better align with the latent distribution obtained from generation without approximations. To derive the target distribution, we use an implicit discriminator that determines whether a given sample originates from generation without approximations:
(()
Dim (zz)
(3)
p(o 2:)
where and ò represent the labels for generation without approximations and that with approximations, respectively.
We then modify the latent to increase the likelihood of the discriminator identifying the sample as generated without approximations. The gradient this objective is computed utilizing Bayes' rule:
Vz. ((z,) = Vz. (- (z.)) - (log p(zz (o) - log p(z)))))
Using Tweedie's formula (Robbins, 1992) and its score-based derivation (Luo, 2022), we approximate Vz, p(g (0) = (z(, t)/at. To minimize L(z,), we update 2, in terms of
és (Zzz. t) = eø (zi, t) + No, Vz, C(z,) ep (z,, t) n Vz. (logp(Ze(o) - logp(z:(o)) (2,, + 7/t (ze( t) EØ (e(z))))
where 7/ the learning rate. Here, (z, t), and én (z,, t) are e predictions performed without approximations and with approximations, respectively.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 Score guidance from unconditional generation
1: Input: weight function w(t), learning rate 7,, condition y, number of total timestep T, timestep offset to. classificr-free guidance strength Wedg
Zr (O
t 2T 4: ~ (0. I) if tor else n = 0
3: unc
5: for fort= T to T 2o..... ........ tor do
6:
c0 Eo (z(4t ta)
7:
2t
une +
une
Pe-dy Pe-de unc + v Sete, n V-millo
D Perform unconditional generation
vutt
8:
Wete) (Zeten U, + to) Wegg (Zetee +
D Apply classifier-free guidanee Conduct score guidance
9:
Eo
+ Whtto me
E0 (zeteo U, to))
Aotio Es + /B440
10:
21 +
ZiHto Vi
11:
n n~ ((0 ift to clse ==
12: end for
13: Output:
For conditional generation, we combine classifier-fret free guidance with Equation and éo (z,, y, t) is obtained as:
Eo (ze v, E + To (ce( - Ez(z,,2 f)),
(5)
where e (2,. y, t) is computed using classfier-free guidance (Ho & Salimans, 2021).
However, since y is encrypted, directly computing 6g (Zz. y, t) without approximations in the ciphertext space infeasible. address this, we use the unconditional score, c4x (z!nc, (), as score guidance, where znee is acquired from unconditional generation without approximations:
(c00 Co (Zi, Zt v, <fe [ze v.t) "/e une (z) t) èo (Zi, y,
An additional advantage of this design is that unconditional generation can be executed rapidly in the plaintext and precomputed offline, allowing the guidance signal be reused during secure inference with negligible computational overhead.
remaining challenge is that using the unconditional score may weaken prompt alignment. Since goal is to generate high-qualij images faithful the conditioning prompt, the guidance of the unconditional score can become detrimental as the denoising process progresses toward = O, where the signal-to-noise ratio highest. To mitigate this effect, gradually reduce the guidance strength by decreasing the learning rate 7/2 as the timestep approaches f = 0. In practice, we adopt % = ryw,, where w, a./āg, Note that we use 7 0.1 for all experiments.
With our score guidance, we observe that image quality is well preserved even when relaxed approxi- mations are applied. As shown in Figure 2, compared using relaxed approximations alone, our method generates sharper and more structurally consistent images, producing feasible number of legs for the sheep and accurately incorporating the baby girl prompt the second column. Additional qualitative comparisons are provided in Figure 5. These results indicate that our framework effectively mitigates the score corruption induced by relaxed approximations.
EXPERIMENTS
In this section, we evaluate our approach on the text-to-image benchmark datasets in secure multi- party computation (MPC) setting. We first briefly describe the experimental setup and evaluation protocol, then compare our method with Stable Diffusion (Rombach et al., 2022) in Section 4. and with CipherDM (Zhao et al., 2024) in Section 4.2. Finally, we provide an ablation study about the learning rate in Section 4.3. To further validate the effectiveness of our approach, method against recent test-time refinement methods in Section 4.4 and evaluate its performance in the few-step sampling regime using consistency models in Section 4.5.
compare

---

Published as a conference paper at ICLR 2026
Table Performance and latency comparison on MSCOCO. FID measures image quality, and CLIP- Score quantifies alignment between generated images and text prompts. Latency (s) is averaged over five runs in a 2PC setting with null prompts, and communication volume (GB) denotes the total communication cost. Latency and communication are measured for single denoising step. while FID and CLIP-Score are computed using approximated models in plaintext due to resource constraints.
Method
FID ())
CLIP-Score (1)
Latency (4)
Comm. Volume (L)
Vanilla Precise-Approx
13.25
0.3254
0.15
13.28
0.3264
3696.45
1122.51
Loose-Approx
15.52
0.3240
3103.32
933.22
Precise-Appros Ours
12.45 14.20
0.3274
3698.13 3105.38
1122.51
Loose-Approx + Ours
0.3259
933.22
Table 2: Quality comparisons on Flickr8k.
Table 3: Ablation study about learning rate (1)).
Method
FID (.)
CLIP-Score (t)
Method
FID (t)
CI.IP-Score (1)
Vanilla
21.17
0.3369
Apprrxx
23.25
0.3347
Precise-Api Apprie
20.85
0,3374
Loose-Approx
23.25
0.3347
Loose-Approx Ours (y=0.005)
21.85
0.3367
Precise- Approx +
19.63
0.3391
Loose-Approx Ours (7=0.110
21.47
0.3372
Loose-Approx Ours
21.47
0,3372
Loose-Approx Ours (y=0.20)
21.25
0.3376
Experimental setting For image generation models, we use CipherDM (Zhao et al., 2024) and pre-trained Stable Diffusion v1.5 (Rombach et al., 2022) without any further fine-tuning. For Ci- pherDM, we use ReLU activations and a DDIM sampler (Song al.) with 50 denoising steps, running under SPU (Ma et al., 2023) with default configurations. For Stable Diffusion, the DDIM sampler (Song et al.) utilized with classifier- free guidance scale of 7.5 (Ho Salimans, 2021) and 50 denoising steps. The latency measurement is conducted using Cryp'Ten (Knott et al., 2021) in a 2PC configuration. For stable approximations, we additionally utilize widely used approximation techniques, detailed in Appendix A. We evaluate performance binary MNIST for CipherDM, and MSCOCO (Lin et al., 2014) and Flickr8k (Hodosh et al., 2013) for Stable Diffusion.
Evaluation protocol We follow the evaluation protocols of Zhao et al. (2024) for CipherDM and Kang al. (2023) for Stable Diffusion. Image quality is assessed using Frechet Inception Distance (FID) (Heusel et al., 2017), while image alignment is measured using the F1-Score for CipherDM and the CLIP--core (Radford al., 2021) for Stable Diffusion. We generate 10,000 images for binary MNIST (5,000 per class), 30,000 for MSCOCO, and 8,000 for Flickr8k. To simulate a local-area network, we employ Linux traffic control with one-way bandwidth of Gbps and 0.3 ms round-trip latency, and report the wall-clock time averaged over five repetitions. For Stable Diffusion, whose full inference latency is significantly high, we measure the latency of a single tatent-diffusion step (excluding the text encoder and VAE decoder) using Cryp'Ten, and generate images for evaluation in plaintext with models where non-linear operations are approximated. When computing latency and communication costs, the maximum operation for softmax stabilization is also calculated in plaintext due to its high computational overhead. For CipherDM, image generation is performed entirely under an MPC setting using SPU (Ma al., 2023). Detailed experimental settings and evaluation protocols are provided in Appendix E.
Model notation 'Vanilla" refers to the original diffusion model. 'CipherDM' represents Zhao et al. (2024) with ReLU activations. Prrecis-Approo indicates the diffusion model using precisely approximated non-linear operations, whereas 'Loose-Approx' denotes the model employing loosely approximated operations as described in Appendix A. '+Ours' represents the corresponding model combined with proposed score guidance from unconditional generation.
7

---

Published as a conference paper at ICLR 2026
Table 4: Performance and latency comparison on binary MNIST using CipherDM (ReLU) under SPU. FID and FI-Score are used to evaluate image quality and the alignment between generated images and conditions (0 1), respectively, Latency (s) and communication volume (GB) are averaged over 5 repetitions.
Method
FID (L)
F1-Score (t)
Latency (L)
Comm. Volume (L)
Vanilla
117.13
0.8575
14.12
CipherDM
316.56
0.0820
430.59
9.12
CipherDM + Ours
242.65
0.6000
442.23
9.14
CipherDM
Ours
Figure 4: Comparison of samples between CipherDM and our method. The first and second rows show samples generated by 'CipherDMM and 'CipherIM+OOus', respectively.
4.1 EVALUATION ON STABLE DIFFUSION
now evaluate our method with Stable Diffusion. Table and 2, relaxing the approximations accelerates inference, reducing latency by 16.04% and communication cost by 16.86% but leads to substantial degradation in both image quality and prompt alignment. Remarkably, when our method is combined, the performance loss is significantly alleviated: for example, MSCOCO the gap in FID between the precise approximation and the loosely approximated model decreases from 2.24 0.98, corresponding of up to 74.17% are observed on Flickr8k). Even with precisely approximated models, our method achieves performance that comparable to or slightly better than the vanilla model. We speculate that this improvement arises since the vanilla model's score can deviate slightly due to random noise. These result implies that our score guidance approach effectively remedies the score corruption. For latency and communication costs, the overhead introduced by our method is negligible. The unconditional score is computed in plaintext, and the guidance computation requires only a single subtraction, addition, and scalar multiplication per denoising step. As result, the overall latency and communication costs remain almost unchanged. A detailed breakdown of the latency and communication costs for each component is provided in Appendix D.
56.25% reduction in quality degradation (similar improvements
4.2 EVALUATION ON CIPHERDM
We also evaluate our method on binary MNIST using CipherDM. For unconditional generation, use the average class embeddings for labels O and As shown in Table 4, our method improves the FID by over 70 compared to the original CipherDM and significantly boosts alignment from 0.0820 to 0.6000. In Figure 4, the samples generated by our method appear much clearer and less noisy than those from CipherDM. Even if the images generated by CipherDM are almost entirely noise, our method can recover clean and coherent digits. These results suggest that our method can effectively improve performance, even when the underlying approximation errors are large.
4.3 ABLATION STUDY FOR LEARNING RATE
We evaluate the sensitivity of the learning rate (n) MSCOCO. In Table 3, the learning rate increases, image quality is improved while maintaining the CLIP-Score. potential concern is that using unconditional scores for guidance might degrade text alignment. However, since the guidance
8

---

Published as a conference paper at ICLR 2026
Table 5: Comparisions with test-time methods.
Table 6: Performance with PCM.
Method
FID (4)
CLIP-Score (1)
Method
FID (L)
CLIP-Score (1)
Loose-Approx
23.25
0.3347
Loose-Approx (50 steps)
23.25
0.3347
Louse-Approx PCM (8 steps)
0.3264
Loose-Approx PAG
36.72
0.3171
Loose Approx PCM (4 steps)
53.80
0.3183
Loose-Approx FKS
24.42
0.3359
Loose-Approx PCM + Ours steps)
27.23
0.3350
Loose-Approx Ours
21.47
0.3372
Loose-Apprex PCM Ours steps)
36.69
0.3264
strength is designed to decrease as the denoising signal grows, text alignment remains unaffected. These results demonstrate that our approach effectively corrects score shifts without compromising text alignment across a broad range of learning rates.
4.4 COMPARISON WITH TEST-TIME APPROACHES
To further validate the effectiveness our approach, we compare against several test-time methods, including saaedddd-sapl---buie refinement (Ahn al., 2024), and test-time scaling strategy (Sing- hal et al., 2025). Under the loose approximation setting, all alternatives perform comparably to or worse than the loose approximation alone (in Table 5), since these methods rely approximated conditional models. Specifically, PAG (Ahn et al., 2024) requires generating mildly degraded sam- ples, but under approximation errors, the degraded samples become severely corrupted, leading to performance degradation. FKS (Singhal et al., 2025), despite using multiple particles, also depends on an approximated verifier model. Thus, verification becomes unreliable and limits its effectiveness. In contrast, our method utilizes unconditional generation and therefore avoids approximation errors.
4.5 EVALUATION WITH CONSISTENCY MODELS
We evaluate our method in the few-shot regime using Phased Consistency Model (PCM) (Wang et al., 2024) with and 8 sampling steps. In Table vanilla PCM suffers from significant performance degradation at lower steps due to approximations. However, integrating our method yields significant improvements, specifically gaining approximately 17 FID at steps and 9 FID steps. These results indicate that our approach provides a practical solution for achieving both efficiency and accuracy in secure inference.
5 RELATED WORK
5.1
CONDITIONAL DIFFUSION MODEL
With the development of diffusion- models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al.; Nichol & Dhariwal, 2021), significant progress has been made in various conditional image generation tasks, such as text-conditioned image synthesis (Saharia et al., 2022) and masked-region image editing (Lugmayr et al., 2022). Various open source models and commercial models such as DALLE (Ramesh et al., 2022), Imagen (Saharia et al., 2022), and GLIDE (Nichol et al., 2022) have contributed to this development. Recently, classifier-free guidance (Ho & Salimans, 2021) simplifies the conditional image generation process by directly incorporating class conditions into the diffusion model, eliminating the need for external classifier. Based on this work, Stable Diffusion (Rombach et al., 2022) has excelled in generating high-quality images efficiently, leading to the exploration of various applications. For test-time scaling, FKSteering (Singhal al., 2025) produces samples that are more aligned with desired preferences by simultancously maintaining multiple particles and iteratively refining them using verifiers. However, secure inference settings, maintaining multiple particles can be computationally expensive, and verification may be unreliable because the verifiers themselves must also be approximated. For sampling guidance, prior approaches (Hong et al., 2023; Ahn et al., 2024) have improved the quality of generated images by refining the scores during inference. These methods encourage models to synthesize more desirable samples by generating undesirable ones and guiding the target scores away from those of undesirable samples. While our approach shares similarities with these methods in promoting the generation of desirable samples, introduces several key differences. First, our focus is generating the samples that correspond

---

Published as a conference paper at ICLR 2026
to images produced without approximations, since generating target samples inherently involves approximation errors.
5.2 PRIVATE INFERENCE
Fully homomorphic encryption (FHE) (Gentry, 2009; Cheon et al., 2017) and secure multi computation (MPC) (Yao, 1986; Evans et al., 2018; Damgård et al., 2019; Goldreich et al., 2019) are widely used in private inference. In contrast to differential privacy (DP) (Dockhorn et al., 2023; Jiang et al., 2025), which focuses on protecting training datasets, these cryptographic approaches aim to protect user inputs from model providers during the inference. CryptoNets (Gilad-Bachin 2016) pioneered the use homomorphic encryption for small neural networks in image classification using polynomial approximations. To exploit the complementary strengths of FHE and MPC, several hybrid frameworks (Juvekar et al., 2018; Reagen et al., 2021; Hao et al., 2022; Pang et al., 2024; Li al., 2024; Lu et al., 2025) apply FHE linear operations and MPC to non-lineat operations. More recently, non-interactive approach (Zhang et al., 2025) has been proposed to further reduce both the computational burden on clients and the associated communication costs. For efficient evaluation of non-linear functions, prior work (Li et al., 2023; Zeng et al., 2023; Zhang et al., 2023) commonly employs piecewise polynomial approximations for activations such as SiLU and GeLU (Hendrycks & Gimpel, 2016), iterative algorithms such Newton or Goldschmidt algorithms (Goldschmidr, 1964) for inverse and square-root computations. For diffusion models, HE-iiffusion (Chen & Yan, 2024) and CipherDM (Zhao et al., 2024) enable preecccyrreriinngggg image generation using FHE and MPC, respectively. However, HE- Diffusion leaves the text prompt unencrypted or unshared, leading potential privacy risks, whereas CipherDM suffers significant performance degradation due to approximation errors. To address these limitations and further accelerate inference, we adopt more relaxed approximations and propose a method that corrects conditional scores by leveraging unconditional scores.
et al.,
CONCLUSION
In this paper, we introduce relaxed polynomial approximations and propose a new framework to mitigate the performance degradation they induce. Specifically, present score-guidance approach that corrects conditional scores by leveraging unconditional generations computed without approximations. Experimental results show that, when combined with approximations, diffusion models can substantially recover performance while incurring only negligible additional costs.
Limitations In this study, we enable diffusion models to generate high-quality images in private inference while substantially reducing latency. However, the absolute latency of single forward pass remains too high for practical deployment in real-world applications, and the performance evaluation of large models is conducted in plaintext. By improving the performance comparable to the original one, believe this research paves the way for practical private inference.
ACKNOWLEDGMENTS
This work was partly supported by Institute for Information & communications Technology Technol- ogy Planning & (valuationnIIIP)) grant funded by the Korea governmen((MSIT) (RS (RS-100.- Artificial Intelligence Graduate School Support Program(KAIST) ment of Vision-Language Model (VLM)-Based Intelligent Video Security Monitoring Technology, No. 2022-0-00984, Development of Artificial Intelligence Technology for Personalized Plug-and-Play Explanation and Verification of Explanation).
Develop-
REFERENCES
Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attention guidance. In European Conference on Computer Vision, pp. 1-17, 2024.
Jimmy Lei Ba. Layer normalization. arXiv preprint arXiv:1607.066550 2016.
10

---

Published as a conference paper at ICLR 2026
Yaojian Chen and Qiben Yan. Privacy-precd preserving diffusion model using homomorphic eneryption. arXiv preprint 400004003 2024.
Jung Hee Cheon, Andrey Kim, Miran Kim, and Yongsoo Song. Homomorphic encryption for arith- metic of approximate numbers. In Advances in Crypholoyy-ATAARRYE 2017: 23rd International Conference on the Theory and Applications of Cryptology and Information Security, Hong Kong, China, December 3-7, 2017, Proceedings, Part 23, pp. 409-437. Springer, 2017.
Ivan Damgård, Valerio Pastro, Nigel Smart, and Sarah Zakarias. Multiparty computation from somewhat homomorphic encryption. In Annual Cryptology Conference, pp. 443-662. Springer, 2012.
Ivan Damgård, Daniel Escudero, Tore Frederiksen, Marcel Keller, Peter Scholl, and Nikolaj Volgushev. New primitives for actively-secure mpe over rings with applications to private machine learning. In 2019 JEEE Symposium on Security and Privacy (SP), pp. 1102-1120. IEEE, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Tim Dockhorn, Tianshi Cao, Arash Vahdat, and Karsten Kreis. Differentially private diffusion models. Transactions on Machine Learning Research, 2023.
David Evans, Vladimir Kolesnikov, Mike Rosulek, et al. A pragmatic introduction to secure multi- party computation. Foundations and Trends@ in Privacy and Security, 2(2-3):70-246, 2018.
Craig Gentry. Fully homomorphic encryption using ideal lattices. In Proceedings of the forty-first annual ACM symposium on Theory of compuling, pp. 169-178, 2009.
Ran Gilad-Bachrach, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Nachrig, and John Wernsing. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In International conference on machine learning, pp. 201-210. PMLR, 2016.
Oded Goldreich, Silvio Micali, and Avi Wigderson. How to play any mental game, or completeness theorem for protocols with honest majority. In Providing Sound Foundations for Cryptography: On the Work of Shafi Goldwasser and Silvio Micali, pp. 307-328. 2019.
Robert E Goldschmidt. Applications of division by convergence. PhD thesis, Massachusetts Institute of Technology, 1964.
Meng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing, Guowen Xu, and Tianwei Zhang. Iron: Private inference on transformers. Advances in neural information processing systems, 35:15718-15731, 2022.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). 1oXx::6606.0815. 2016.
arXiv preprint
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffuusin guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Micah Hodosh, Peter Young, and Julia Hockenmaier. Framing image description ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853-899, 2013.
11

---

Published as a conference paper at ICLR 2026
Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1462-7471,
Tanqiu Jiang, Changjiang Li, Fenglong Ma, and Ting Wang. Rapid: Retrieval augmented training of differentially private diffusion models. In The Thirteenth International Conference on Learning Representations, 2025.
Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. {GAZELLE): A low latency framework for secure neural network inference. In 27th USENIX security sympostum (USENIX security 18), pp. 1651-1669, 2018.
Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10124-10134, 2023.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980. 2014.
B. Knott, S. Venkataraman, A.Y. Hannun, S. Sengupta, M. Ibrahim, and L.J.P. van der Maaten. Crypten: Secure multi-party computation meets machine learning. In arXiv 2109.00984 2021.
Dacheng Li, Hongyi Wang, Rulin Shao, Han Guo, Eric Xing, and Hao Zhang. Mpeformer: Fast, performant and private transformer inference with mpc. In The Eleventh International Conference on Learning Representations, 2023.
Zhengyi Li, Kang Yang, Jin Tan, Wen-jie Lu, Haoqi Wu, Xiao Wang, Yu Yu, Derun Zhao, Yancheng Zheng, Minyi Guo, et al. Nimbus: Secure and efficient two-party inference for transformers. 2024.
Advances in Neural Information Processing Systems, 37:21572-211
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Wen-jie Lu, Zhicong Huang, Zhen Gu, Jingyu Li, Jian Liu, Cheng Hong, Kui Ren, Tao Wei, and Wenguang Chen. BumbleBee: Secure Two-party Inference Framework for Large Transformers. In 32nd Annual Network and Distributed System Security Symposium, NDSS 2025. The Internet Society, 2025.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11461-11471, 2022.
Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXh::208..19970 2022.
Junming Ma, Yancheng Zheng, Jun Feng, Derun Zhao, Haoqi Wu, Wenjing Fang, Jin Tan, Chaofan Yu, Benyu Zhang, and Lei Wang. SecretFlow-SPP A performant and User-Frienddy framework for Diraac-Prreerving machine learning. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pp. 17-33, Boston, MA, 2023. USENIX Association. ISBN 978- 1999133--5--
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. In International conference machine learning, pp. -171- PMLR, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pp- 16784-16804. PMLR, 2022.
Qi Pang, Jinhao Zhu, Helen Möllering, Wenting Zheng, and Thomas Schneider. Bolt: Privacy- preserving, accurate and efficient inference for transformers. In 2024 IEEE Symposium on Security and Privacy (SP), pp. 4753-4771, IEFF, 2024.
12

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Brandon Reagen, Woo-Seok Choi, Yeongil Ko, Vincent T Lee, Hsien-Hsin Lee, Gu- Yeon Wei, and David Brooks. Cheetah: Optimizing and accelerating homomorphic encryption for private inference. In 2021 IEEE International Symposium on High- Performmeee Computer Architecture (HPCA), pp. 26-39. IEEE, 2021.
Herbert E Robbins. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, pp. 388-394. Springer, 1992.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp.
2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36444 2022.
Raghav Singhal, Zachary Horvitz, Ryan Techan, Mengye Ren, Zhou Yu, Kathleen Mckeown, and Rajesh Ranganath. A general framework for inference-time scaling and steering of diffusion models. In International Conference on Machine Learning, pp. 55810-55821 PMLR, 2025.
Jascha Sohl-Dickstein, Fric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning. pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations.
Fu- Yun Wang, Zhaoyang Huang, Alexander Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Keqiang Sun, Weikang Bian, Guanglu Song, Yu Liu, et al. Phased consistency models. Advances in neural information processing systems, 37:83951-8444 2024.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pp- 3-19, 2018.
Andrew Chi-Chih Yao. How to generate and exchange secrets. In 27th annual symposium on foundations of computer science (Sfcs 1986), pp. 162-167. IEEE, 1986.
Wenxuan Zeng, Meng Li, Wenjie Xiong, Tong Tong, Wen-jie Lu, Jin Tan, Runsheng Wang, and Ru Huang. Mpevit: Searching for accurate and efficient mpe-friendly vision transformer with heterogeneous attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5052-5063, 2023.
Jiawen Zhang. Xinpeng Yang, Lipeng He, Kejia Chen, Wen-jie Lu, Yinghao Wang, Xiaoyang Hou, Jian Liu, Kui Ren, and Xiaohu Yang. Secure transformer inference made non-interactive. In NDSS, 2025.
Yuke Zhang, Dake Chen, Souvik Kundu, Chenghao Li, and Peter A Beerel. Sal-vit: Towards latency efficient private inference on vit using selective attention search with a learnable softmax approximation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5116-5125, 2023.
Xin Zhao, Xiaojun Chen, Xudong Chen, He Li, Tingyu Fan, and Zhendong Zhao. Cipherdm: Secure three-party inference for diffusion model sampling. In European Conference on Computer Vision, pp. 288-305, 2024.
13

---

Published as a conference paper at ICLR 2026
A APPROXIMATION DETAILS
For the polynomial approximations of non-linear functions such as SILU, GeLU (Hendrycks & Gimpel, 2016), and the exponential function in softmax, we determine the coefficients by minimizing the mean squared error (MSE) over a targeted input interval. To select this interval, we perform a discrete search to identify the boundary that minimizes the MSE within the original segment. For example, in SiLU, the lower bound of -5.4 is obtained by treating the boundary as a hyperparameter and searching over {-7.9, -78, ... 3.9}, minimizing the MSE in the range [' 8, 4). For numerical stability during secure inference, we also apply stabilization techniques to GroupNorm (Wu & He, 2018) and softmax.
SiLU In BumbleBee (Lu et al., 2025), SiLU is partitioned into four input segments and approxi- mated by low-degree polynomials as given by
0,
if < -8 if 8 -4
a20² + axxt 1 + 00,
PocciseSiiU((s
6 byx b22² + b1 + bo
if - 4
if 2 4,
where d2 -00000000000544555550000000000.0 00.8199660021555476 0gg 0.30675411139961155 by 0.00027433766353465, bu = 011113046708117 by =
b = 0.5, bo 0.008500640255355551. For the simplified version, SiLU reduced three imput segments and expressed as
0. Cg + c6x + cxx2 + c2 + C1 + cD. 2,
if 5.4
(oose-SiLU(a)) =
- 5.4 x < i 2 4,
where 0g 00. 000.0000528800334, Gg 0.00099933016,, -.00118556655 0.228920817, = 0.500011828, co 0.00852801754.
GeLU BumbleBee, GeLU partitioned into four input segments and approximated by low- degree polynomials as given by
0,
if
0zr's + aza² + +axx + ao.
if - < < -197
(re)
b + bar + b2x² + by + bo:
if - 1.97 < if 2,
where 03 = 0.011003411-000066557, a2 = 0.1180761195118119, a1 0.442265811511983858
an 0.50550011199708177,, by 0.00180674626001111 bu = "0.0376882003365904,
b/ 0.3603292692788622 b = 0.5, bo = 0.008526321 1.006526:215410380 5410380. For the simplified version, GeL.U
partitioned into three input segments and approximated by
0.
if f -21 if - - < x < if 2 3,
Loose-GeLU(z)
c,z o2 + c1 + co.
where C4 = -0.01625818, 02 0.30074739, 0.49693733, co 0.03222096.
Exponential in softmax Similar to BumbleBee, the exponential function is partitioned into two input segments and approximated by
0,
if 14
Precise-Exp(z)
(11 + 23
2
if 2 -14.
Since the input the exponential function in the softmax is always non-positive, we further approxi- mate the second segment with a fourth-degree polynomial,
50,
if -4.8 c/ + czx + c2x² + ce co. if 4.8 < x S 0,
Loose-Exp(zz)
14

---

Published as a conference paper at ICLR 2026
where cд 0.00488676, сy 0.06755453, c2 0.36066246, c 0.91766503, cu 0.98805643.
In CipherDM (Zhao et al., 2024), the exponential function is approximated with Chebyshev polyno- mial after mapping from [-14, to [-1, as:
0, E Ei=o.... Ca
if < 14
CipherDM-Expp2))
if - x 5
where C7 0.6377824, 0.66811776 C5 -0.41698096, C4 0.28150564, C2 = 0.09208474,cc c1 = -0.0238906, Co 0..01111495.
0.27598784,
GroupNorm For numerical stability, in GroupNorm and LayerNorm (Ba, 2016), we scale down the variance of the input tensor by a sufficiently large factor before normalization using the fact that x-mean(x)) x/d-meulix/d) for any non-zero d. To determine d, we increase d as the number deviation(x)
deviation(x/d]
of channels the group decreases, leveraging the fact that variance is proportional the number of samples. Specifically, in the text encoder, we set d = 8 for LayerNorm. In the latent diffusion model, we use d = max( / (64 64 - 30) 1) for GroupNorm, where d, denotes the dimension of for LayerNorm. In the decoder of the compression model, we apply d (512 512- 256 d for GroupNorm.
the channels of each group,
Softmax In softmax, subtract the maximum input value before computing softmax, as exp(xz) exp(a, FFmec)
E, exp(a,) Σ, ( for 2'mxx = maxj zi.. Since the model processes 4096 visual tokens in Stable Diffusion (Rombach al., 2022), we divide the input tensor by the square root of the number of tokens after applying the exponential but before applying the reciprocal. Thus, we compute the (afd:/=mmm/// where is the number of tokens. (esp((rr /a))"
softmax as: Σ, exp(as--aa Fmd)
exp(æ; [;-a)
B EXPERIMENT UNDER WIDE AREA NETWORK
Using Linux traffic control, we simulate Wide Area Network (WAN) with 400Mbps bandwidth and a 4ms ping time to evaluate our method on CipherDM. As shown in Table 7, when the unconditional score is computed during inference, the latency overhead introduced by our method is only 1. 1.12% Moreover, because the unconditional score can be precomputed offline, excluding this cost reduces the latency overhead to just 0.02%, which is negligible relative to the total runtime.
Table 7: Performance and latency comparison on binary MNIST using CipherDM (ReLU) under SPU. FID and F1-Score are used to evaluate image quality and the alignment between generated images and conditions (0 or 1), respectively. Latency (S) and communication volume (GB) are averaged over 5 repetitions under Wide Area Network (WAN).
Method
Latency (L)
Comm, Volume (L)
Vanilla
14.12
CipherDM
1250.87
9.11
CipherDM + Ours
1265.26
9.13
C IMPACT ANALYSIS OF ENCODER AND DECODER APPROXIMATIONS
To investigate the individual contributions of text encoder and VAE decoder approximations to the overall performance, we conduct experiments by replacing them with exact computations while maintaining approximations in the latent diffusion model (LDM). As shown in Table 8, eliminating these approximations reduces the quality degradation by approximately 1.3 FID. Notably, under this configuration, proposed method fully recovers the performance of the vanilla model. This result underscores that our score guidance effectively addresses the predominant approximation errors that arise within the latent diffusion component.
15

---

Published as a conference paper at ICLR 2026
Table 8: Ablation study on the impact of exact text encoder and VAE decoder computations on MSCOCO. FID (U) and CLIP-Score (f) are reported for quality and alignment assessment.
Method
FID (4)
CLIP-Score (t)
Vanilla
13.25
0.3254
Loose-Approx
15.52
0.3240
Loose-Approx + Exact-Enc-Dec
14.22
0.3257
Loose-Approx + Exact-Enc-Dec Ours
13.10
0.3269
D COMPUTATIONAL PROFILING
We analyze the computational overhead of each module to identify the primary bottleneck. Table presents the latency and communication costs measured 2PC setting. While the text encoder's contribution is negligible and the VAE decoder shows moderate costs, the latent diffusion model (LDM) represents the majority of the total latency. Given that the LDM requires iterative evaluation, improving its approximation efficiency is critical for practical deployment. These observation justifies our strategy of applying loose approximations specifically to the LDM to achieve substantial speedups.
Table 9: Latency (8) and communication cost (GB) breakdown by component in 2PC configuration. Measurements were conducted to evaluate the relative overhead of each module.
Module
Latency (1)
Comm. Volume (1)
Text-Encoder
40.34
7.76
'AE-Decoder
2668.66
763.40
Latent Diffusion (1-step)
3696.45
1122.51
E EXPERIMENTAL SETUP
We now provide more detailed experimental settings for the evaluation protocol and inference details.
Evaluation protocol We follow the evaluation protocols of Zhao et (2024) for CipherDM and Kang et al. (2023) for Stable Diffusion. To compute the Fl score, we fine-tune ResNet-18 (He et al., 2016) pre-trained on ImageNet (Deng et al., 2009) by replacing and training only the classifier head, while freezing all other parameters. The model is fine-tuned on the binary MNIST training set (digits and 1) for epochs using the Adam optimizer (Kingma, 2014) with a learning rate of 0.001. The classification accuracy of the fine-tuned ResNet-18 18 is nearly perfect the test set, ensuring reliable Fl evaluation. To compute the FID score on binary MNIST, we use only the training set containing digits 0 and 1. For latency estimation in CipherDM,, 480GB RAM for Ci. For Stable Diffusion, we utilize Intel(R) Xeon(R) Gold 6226R CPU, 256GB RAM, and NVIDIA RTX A6000 GPU (48GB VRAM).
Intel(R) Xeon(R) Gold 6226R CPU and
Inference details We adopt learning rate of 1 = 0.1 for all experiments. Since the architecture of CipherDM uses class embeddings for conditioning, we use the mean of all class embeddings during unconditional generation.
F THREAT MODELS
We follow a standard threat model commonly adopted in private inference literature, assuming semi- honest adversaries and computationally bounded participants. In our implementation, we adopt two representative MPC frameworks: CrypTen (Knott et al., 2021), which operates in 2-party setting, and SPU (Ma et al., 2023), which uses a 3-party setting. Our system design and empirical evaluations aligned with the operational guarantees of these frameworks.
16

---

Published as a conference paper at ICLR 2026
Precise-Appror
Loose-Approx
Loose-Approx Ours
An image o train that going
large brown stufted ariinal laying
white vase fille with purple grewn
Doonway vie into
riding siis covered
Precise-Approx
Loose-Approx
Loose-Approo Ours
pirk papar scasons
badk of waler.
canial with blumed backgroound.
bear laylng under
cupboard of knivea sitiing on
Market
Figure 5: Comparison of text-to-image samples generated by precisely approximated model (Precise- Approx), loosely approximated model (Loose-Approx), and loosely approximated model with our method (Loose-Approx Ours).
G MORE EXAMPIEE
Below, provide visualizations of generation examples for extended qualitative assessment in Figure When the approximation is relaxed, the model struggles to produce fine-grained components, such fingers the precise shapes of moving objects, and often yields blurred images compared with the precisely approximated model. In contrast, when combined with our method, the model successfully generates clear fingers, well-defined object contours, and sharp images, demonstrating that our approach effectively mitigates the score corruption introduced by relaxed approximations.
17


Reviews:
[{"review_id": "iDFivekGOJ", "reviewer": "Reviewer_f36X", "summary": "The paper targets private inference for conditional diffusion under MPC/HE by replacing non-polynomial ops (SiLU, GeLU, exp/softmax) with piecewise polynomial approximations. Relaxing these approximations lowers latency but corrupts the conditional score; the core idea is to use a plaintext, high-fidelity unconditional score (computed without approximations) to correct the encrypted conditional score during sampling. The method derives a guidance update, yielding a simple correction term. Experiments on Stable Diffusion (MS-COCO, Flickr8k) and CipherDM (binary MNIST) show reduced degradation.", "questions": "1. **Iterative rectification within a step**\n    - Since the guidance explicitly rectifies the score, can it be applied iteratively within the same time step? If so, does increasing the number of inner refinements (k=1,2,3,…) improve fidelity in a cost-aware manner.\n\n2. **Few-/one-step regimes**\n    - As diffusion/flow models move to few-step or even one-step samplers (e.g., consistency models, meanflow), how does the method specialize? In a single-step setting, is there a principled way to form the unconditional reference (score/velocity) and apply your correction so that it still yields a measurable gain?\n\n3. **Budgeted trade-off vs. more steps**\n    - The method is a test-time scaling that adds overhead. Under a fixed latency/communication budget, how does guidance compare to simply adding a few extra sampling steps (or slightly tightening the polynomial approximation) without guidance?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Clear presentation and reasonable motivation**\n    - The paper motivates the problem well: because the text condition is inaccessible during secure inference, the authors refine the conditional path by injecting a clean, uncorrupted unconditional score as a corrective signal.\n\n2. **Simple, architecture-agnostic mechanism**\n    - The method is a small, drop-in change to the sampler and is model-agnostic. And it only introduces essentially two scalar knob: the guidance strength and the learning rate.\n\n3. **Good experimental design with clear effectiveness** \n     - The experiments probe both sides of the trade-off (latency/communication vs. fidelity) across multiple approximation regimes and datasets, and consistently show that the guidance improves quality to relaxed polynomial approximations.", "weaknesses": "1. **Minor quality gains at similar cost** \n    - Relative to no score guidance, the improvements are modest, and under the loose approximation, even with the proposed guidance, the generation quality still lags behind the precise approximation without guidance at roughly comparable latency (cf. Tables 1–2). \n2. **Incremental contribution; missing comparisons to alternative test-time fixes**\n    - The method functions as a test-time refinement/scaling of the sampler, and the “implicit discriminator” view echoes prior discriminator-style interpretations of likelihood models (albeit those are usually applied during training rather than at test time). To justify the novelty and practical value, it would be helpful to compare this approach with other lightweight test-time or post-sampling corrections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qz4oGp9zEy", "reviewer": "Reviewer_MYHs", "summary": "This paper proposes a novel add-on to an existing privacy-preserving inference method (CipherDM) by addressing the issue of high latency and low fidelity of generated images. The key contribution is that the proposed method leverages the score guidance from the unconditional generation, which provides a higher fidelity signal for image generation. The results show that the image quality has increased after applying the new method on top of CipherDM.", "questions": "1. Would it be possible to apply the proposed method in this paper on traditional structures such as DDPM or DDIM? What will be the effect? \n\n2. What is the difference or pros and cons between using private inference or Differentially Private Diffusion Models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1, The paper is well written and easy to follow.\n\n2. The idea of using the unconditional generation as a guidance signal is interesting and novel. \n\n3. Through experiments, the authors have shown that the new add-on to the CipherDM can effectively increase the fidelity of images.", "weaknesses": "1, All experiments are conducted on a pretrained Stable Diffusion v1.5 Model, while the CipherDM paper was mainly evaluated on DDPM and DDIM. It might be better to address the transferability issue of the proposed method across different diffusion architectures. \n\n2. In parallel with private inference, another line of research called differentially private diffusion models are serving for the same purpose of privacy preservation. Recent works such as DPDM (https://arxiv.org/abs/2210.09929) and RAPID (https://iclr.cc/virtual/2025/poster/28006) are also trying to provide privacy guarantees to diffusion models. From both papers, I can see that their FID scores on MNIST dataset are much lower than this work or CipherDM. It might be more helpful to address this issue by either comparing their performances through experiments under the same setting or addressing their differences in a separate subsection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oFVxLQaduJ", "reviewer": "Reviewer_tfMV", "summary": "The paper studies efficient implementation of inference for diffusion models (IDM) under secure multi-party computation (MPC). A computational challenge encountered in previous work (CipherDM, Zhao et al., 2024) is that inference requires the computation of non-linear functions on the ciphertext, which is known to be computationally intensive. CipherDM addressed this issue by switching to polynomial approximations of the non-linear function, and so improving the scalability.\n\nThis paper revisits CipherDM with a coarser polynomial approximation, showing improved efficiency at the cost of a further deterioration in output quality. To mitigate the drop in output quality, the authors propose to augment the inference process by using unconditional and exact generation in plaintext to “steer” the diffusion process. The key idea is that the unconditional generation has no sensitive content, and so this signal can be computed fast and exactly in plaintext, and can consequently be used to guide the diffusion process to correct for errors induced by the polynomial approximation at a low computational overhead. Experiments confirm that the augmented signal can compensate for much of the approximation error, even to the point of beating the exact baseline in some cases.", "questions": "I find the paper’s topic of efficient secure inference for diffusion models interesting, its proposed technique clever and its writing of high quality. The empirical work is clean and appears reproducible. My only concern is related to fine-tuning as an alternative to the “signal steering” approach proposed, which I think is dismissed with insufficient motivation. As things stand, I lean towards acceptance, and feel more inclined to raising my score than lowering it.\n\nI have some questions. As they are a few, feel free to give brief answers where you see fit.\n1. My understanding of your approach is that you use the signal from the “uncondtional+exact” latent diffusion model to steer the approximated latent diffusion model to correct for approximation errors. Nevertheless, from Figure 3 it seems like your text encoder and decoder are also approximated. Did you try investigating how much of the drop in image quality is due to these particular approximations for your approach? If your technique corrects for most of the approximation error in the latent model, then I would expect the performance to increase if the encoder/decoder are exact, at least in the case of the coarse approximation.\n2. Related question: with no approximations, how much (roughly) of the latency is driven by the encoder/decoder step from Figure 3? My understanding is that Table 1 and 2 do not measure their influence, so I am curious how important they are to approximate.\n3. What is the cost for instead fine-tuning the DM with polynomially-approximated activations driven by? Is it only the cost of fine-tuning a “normal” DM, or is there some additional cost that is imposed from the setting of secure inference?\n4. You observe in Table 1 and 2 that your signal steering can not only compensate for the approximation error, but improve on the baseline if the approximation is precise enough. You hypothesize in Section 4.1 that this improvement could be driven by that your steering has a stabilizing influence on the process. Did you consider using your steering technique on the *exact* baseline itself? It would be interesting to see if the 'vanilla' model + steering also led to some improvement.\n5. In the related work, you mention sampling guidance (Hong et al., 2023; Ahn et al., 2024). Could your idea of \"signal steering\" with the exact model in plaintext be combined with their techniques, or are they incompatible? Can \"sampling guidance\" be implemented without additional computation on ciphertext?\n6. While it might be computationally expensive in general, did you consider testing fine-tuning of the DM for e.g., the MNIST dataset? E.g., fine-tune CipherDM and see how it compares with the results in Table 4?\n7. Is the last sentence of the caption to Table 1 related to FID and CLIP-Score important for interpreting the accuracy results (computations being done in plaintext)? I would expect the reported image quality to be independent of this detail.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written with a clear problem statement supported by nice figures.\n2. The key idea of using unconditional sampling in plaintext space to (with small overhead) inform the generation process is natural and clever.\n3. The problem is well-motivated and fits the scope of ICLR.\n4. The experimental results are strong, and suggest that activation functions can be polynomially approximated at little to no cost in image quality (Table 1 and 2).\n5. The empirical setup is given in enough detail (including an attached code artifact) to be reproducible.", "weaknesses": "1. The alternative of fine-tuning the DM with poly-approximated activations is not considered in depth. Since the main contribution of the paper is to make secure inference for DM more efficient, it would be fitting to have a more extensive discussion of this alternative approach and its accuracy. Line 212-214 correctly makes the point that the corresponding fine-tuning of large DMs is expensive, but one could nevertheless imagine large industry actors training DMs for the express purpose of enabling high-performance secure inference. If there is a gain in output accuracy from fine-tuning, they may find the associated cost worth taking on. See later questions.\n2. (Conditioned on the validity of the previous point) If fine-tuning the DM with poly-approximated activations can achieve better accuracy than the proposed method in this paper, then the impact of the contribution is reduced. The authors improve the accuracy/efficiency trade-off for MPC-IDM only in the particular case where a diffusion model is not trained with MPC in mind, and is then “retrofitted” for the purpose. Nevertheless, I concede that (1) this might be an important case, and (2) in the case of precise approximation the empirical work suggests that fine-tuning the model may be excessive.\n3. A minor point. While it is mentioned that CipherDM (Zhao et al., 2024) also employed polynomial approximation of activation functions (line 49) to improve performance for secure inference, the details of their approximation are (as far as I can tell) not discussed. Section 3.1 discusses polynomial approximations (“precise” and “loose” variants), but does not relate this to CipherDM. For completeness, it could be nice to include their approximation scheme in the appendix. It would also be useful from the point of view of having more interpretability of Table 4, here the signal steering, while helpful, appears to still lag far behind the baseline---is it due to the coarseness of the approximation in CipherDM? I don't think this can be inferred from the paper currently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)