Pin the Tail on the Model: Blindfolded Repair of User-Flagged Failures in Text-to-Image Services
Authors: Gefei Tan
Ali Shahin Shamsabadi
Ellen Kolesnikova
Hamed Haddadi
Xiao Wang
Venue: NeurIPS 2025 poster ()
Keywords: Model Editing
PPML
Secure Computation
Diffusion Models
Cryptography
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Diffusion models are increasingly deployed in real-world text-to-image services. These models, however, encode implicit assumptions about the world based on web-scraped image-caption pairs used during training. Over time, such assumptions may become outdated, incorrect, or socially biased--leading to failures where the generated images misalign with users' expectations or evolving societal norms. Identifying and fixing such failures is challenging and, thus, a valuable asset for service providers, as failures often emerge post-deployment and demand specialized expertise and resources to resolve them. In this work, we introduce $\textit{SURE}$, the first end‑to‑end framework that $\textbf{S}$ec$\textbf{U}$rely $\textbf{RE}$pairs failures flagged by users of diffusion-based services. $\textit{SURE}$ enables the service provider to securely collaborate with an external third-party specialized in model repairing (i.e., Model Repair Institute) without compromising the confidentiality of user feedback, the service provider’s proprietary model, or the Model Repair Institute’s proprietary repairing knowledge. 
To achieve the best possible efficiency, we propose a co-design of a model editing algorithm with a customized two-party cryptographic protocol.
Our experiments show that $\textit{SURE}$ is highly practical: $\textit{SURE}$ securely and effectively repairs all 32 layers of {Stable Diffusion v1.4} in under 17 seconds (four orders of magnitude more efficient than a general baseline). Our results demonstrate that practical, secure model repair is attainable for large-scale, modern diffusion services.

Pin the Tail on the Model: Blindfolded Repair of User-Flagged Failures in Text-to-Image Services
Gefei Tan'
Ali Shahin Shamsabadi' Brave Software
Northwestern University
Ellen Kolesnikoval Decatur High School
Hamed Haddadi
Xiao Wang
Brave Software & Imperial College London
Northwestern University
Abstract
Diffusion models increasingly deployed in real-world text-to-image services. These models, however, encode implicit assumptions about the world based on web- scraped image-caption pairs used during training. Over time, such assumptions may become outdated, incorrect, or socially biased-leading to failures where the generated images misalign with users' expectations evolving societal norms. Identifying and fixing such failures is challenging and, thus, a valuable asset for service providers, as failures often emerge post-deployment and demand specialized expertise and resources to resolve them. In this work, we introduce SURE, the first end-to-end framework that Sec REpairs failures flagged by based services. SURE enables the service provider to securely collaborate with an external third-party specialized in model repairing (i.e., Model Repair Institute) without compromising the confidentiality of user feedback, the service provider's s proprietary model, the Model Repair Institute's proprictary repairing knowledge. To achieve the best possible efficiency, we propose co-design of model editing algorithm with a customized two-party cryptographic protocol. Our experiments show that SURE is highly practical: SURE securely and effectively repairs all 32 layers of Stable Diffusion vI.4 under 17 seconds (four orders of magnitude more efficient than a general baseline). Our results demonstrate that practical, secure model repair attainable for large-scale, modern diffusion services.
of diffusion-
Introduction
17 are helping millions of users to create images from textual prompts 45). These services typically powered by test- when trained on billion-scale datasets of image-caption pairs seraped from the web. However, diffusion models
growing number real-world services 26
29
25
40 21
24 28
to-image diffusion models [9 33), which generate high-quality images 45 7
implicitly encode the knowledge and assumptions present in their training data [15] which then appear again during image generation. This can lead unintentional failures: although the generated image may be high quality and technically accurate, it can still misalign with users' values and expectations. For example, diffusion models might retain outdated incorrect information (e.g., the identity of country's 's president or celebrity's 's hairstyle). More importantly, diffusion models may encode harmful stereotypical assumptions about professions into their parameters. For example, when given the prompt "A photo of CEO", the commercial image generation services predominantly generate images of men only 4% of outputs depict women (30]
31 3
38 g
"Contributed equally as co-first authors. I Work done during internship Northwestem University.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

User
Textual prompt (1)
Service Provider
ModelReppir Institute Z
Failure & Model
G
Repair Detabase
Diffusion Model
Secure Model Repair
FallureFFkk
Image
Repaired Dillusion Model
Outdlaned, incorrect socially biased assumption
Fallure
foodback
Repair Aigorithm
Block diagram of SURE. A service provider S deploys a diffusion model to generate images (services) in response textual prompts (user queries) (I) & 2), When user U notices a failure in S's services-due the outdated, incorrect discriminative assumption- (3), 24 provides feedback to S (+). S then collaborates with a ModelRepair Institute to securely repair the model (O) through cryptographic protocols that preserve the confidentiality of users' feedback, service provider's 's model and institute's proprietary repairing knowledge. Finally, the repaired model is returned only to the service provider (6).
Figure I:
When model failure happens in practice, users typically discover these failures and provide feedback on the current behavior of models to the service providers [IO]. However, it is challenging for service providers to incorporate feedback and repair their models for several reasons. First, these unintentional failures emerge over time [3]] as world knowledge or societal norms evolve. Second, repairing diffusion models usually requires substantial expertise and resources 4 which service providers, especially start-ups, lack. One possible solution to address this problem is for the service provider to share its model and user feedback with an external institution specializing in model repair) who can repair the failure. However, this approach raises significant concerns for all parties involved. Sharing the model compromises its confidentiality, undermining the commercial value of service provider's s image-generaae regulations such as the GDPR [4]] Mcanwhile, the repair institution is reluctant to disclose its repair techniques in order to protect its intellectual property.
service. Sharing user feedback is also not permissible due to privacy
Our Work. We address these challenges by introducing two-party secure repairing based on user feedback. We propose SURE (Figure I.. secure framework that enables service provider and an external model repair institution to collaboratively repair the service provider's diffusion model using users' feedback and the institution's 's repair expertise while remaining mutually blindfolded. To ensure that the users' feedback, the provider's proprietary model, and the expert's repair recipe all remain confidential, SURE leverages secure two-party computation (2PC) techniques 46 16). which allow two parties to jointly compute any function without revealing anything about their private data beyond the function output. Directly computing existing knowledge editing algorithm [30] [5] 45 in 2PC is theoretically feasible, but becomes completely unrealistic in practice due to the high computation cost of both 2PC and knowledge editing algorithms. Instead, take co-design approach to jointly optimize both the machine learning and cryptography components. SURE targets and updates only tiny fraction of narameters-namely. the keys and values of cross-attention layers- with a crypto-friendly repair formula. Our design enables each party to shift expensive operations offline, allowing
design lightweight, customized cryptographic protocol top of it.
Our protocol consists of (I) a small 2PC circuit that privately matches the user feedback to the most relevant fix and (2) an oblivious-transser-bbe based protocol [32] that securely delivers the corresponding fix. Our protocol completely avoids matrix operations inside 2PC, and the cryptographic overhead remains constant regardless of the number of layers repaired. Our end-to- end-to-end secure repair framework is highly efficient and scalable: our experiments show that a service provider can use SURE to repair all 32 layers of their Stable Diffusion v1.4 34 in collaboration with Model Repair Institute in under 17 seconds, whereas an optimized baseline protocol needs over 100 hours.
In summary, we propose the first secure repairing framework that enables users to control the model's ss behavior over time and enables service providers to ensure continued alignment with social expectations post-training without retraining, We highlight the following contributions:
We initiate the study of an important and emerging problem model repair while protecting the security of the model, data, and the repairing knowledge. We formulate the security and utility requirements needed in real-world applications.
httpe://humanfeebbcck....

---

Although generic cryptographic protocols can be used to support this task, their efficiency is completely unacceptable for realistic applications. To this end, we co-design crypto-friendly editing algorithm and customized 2PC protocol such that the editing algorithm is as effective as state-of-the-an model repair approaches while minimizing the protocol cost when executed using our optimized cryptographic protocol.
We implemented our protocol and baseline protocol using generic 2PC. We tested their perfor- mance for repairing Stable Diffusion v1.4. We observed 4 orders of magnitude improvement in runtime compared to the baseline, bringing secure model repairing from merely concept to something that can practically be deployed on modern models.
2 Notations and Preliminaries
Notations. We lowercase bold letters like denote column vectors and uppercase bold letters like W to denote matrices. We write [n] to denote the set (1, 2....... n3.. We use consistent notation for values in the diffusion model architecture, defined in the next few paragraphs.
Diffusion Models [9 (39] are a class of generative models that have recently emerged as the SOTA in image generation. Inspired by non-equilibrium thermodynamics, diffusion models use fixed algorithm to incrementally add random noise to images (or other data), and then learn how to reverse this process. The learned model is then used for image generation. Diffusion models have not always been the SOTA in image generation; prior to diffusion models, GANs were the most promising image generation models [I].. However, compared to GANs, diffusion models offer multiple advantages that lead to better results [2]. Diffusion models use more stable loss metrics than GANs. Additionally, because diffusion models generate images over a series of timesteps, their task is easier than that of GANs, which do it in one pass.
where the diffusion process is guided by a user-provided text prompt that is embedded and injected into the cross-attention layers of the model. Formally, we consider diffusion model M that generates images by denoising Gaussian sample xT over T time steps using a neural network (x,, t, c). where c is a conditioning signal derived from the text. The text prompt is first tokenized and processed by text encoder, which outputs a sequence of token embeddings {c, V._ where c; E R that represent the semantic content of the input text. Let C = (c.,..., cr. E Rext denote the resulting matrix of text embeddings. At every aoossattt layer. these embeddings are linearly projected into K W e R and V = WW E Rx using learned key and value projection matrices Wx K E k and Wv E IR" respectively. Next, the key K is multiplied by a query Q E Ra*k that represents the current image's visual feature. The cross-attention mechanism computes an attention map and weighted value
In this work, we focus on text-to-image diffusion models 33
36
27
18 35
output: M = softmax
QK and O = MV. The output O guides the visual features based on vm
the semantic content of the text prompt.
Diffuusion Model Editing aims to remove various biases from diffusion models and has become increasingly important as these models gain widespread adoption. One way is done is by adjusting various aspects of the training process to limit bias; this can include altering the loss function (37 or debiasing the training dataset [23] Fine-tuning existing diffusion models is perhaps more realistic approach, as biases can become apparent after training. To do this, a small fraction of the weights in the diffusion model are updated to fix specific problem. This can be done by editing the text encoder [44), or by directly editing the diffusion model [13 30). We focus on fine-tuning after training in this paper, as this ensures models can be updated needed and do not need to be retrained.
Oblivious Transfer (OT) is a fundamental cryptographic primitive essential for secure computation protocols 32). In 1fout-of-n OT, sender possesses messages (mı,.... m,), and receiver selects an index i E [m]] to retrieve m, without revealing i to the sender. Simultancously, the receiver gains no information about the other messages m, for + i. This ensures that the sender remains oblivious to the receiver's 's choice, and the receiver learns only the selected message.
Secure Two-Party Computation (2PC) 4b 16 enables two mutually distrustful parties, each holding private inputs, to jointly compute a public function without revealing any information beyond the output. We consider 2PC in the presence of static semi-honest adversaries, where parties follow the protocol but may attempt to learn additional information from the protocol execution transcript. The ideal functionalities of lfout-of-n OT and 2PC are presented in Appendix B
3

---

3 Problem Description
Parties and Trust Assumptions, We consider setting including three parties as illustrated in Figure [ a service provider S that offers diffusion- image generation services, users U who query the service and provides feedback when observing service failures, and a model repair institute I, which specializes in repairing model failure and collaborates with S to repair its model.
In this setting, we make the following trust assumptions in our threat model:
Users U query the image generation service with their textual prompt and receive images. 2/ discovers failures as they use M--ased services of S. ll's flagged failures are because of the fact that M acquires knowledge within their training data [30 which become outdated, incorrect and harmful over time. For instance, for the prompt "A photo of CEO", only 4% of generated images (with random seeds) contain female figures [30] This feedback should only be visible to S.
A Service Provider S trains the text-to- text-to-image diffusion model M on huge amounts of web-scraped image-caption pairs, and provides image generation services using M. 8 wants to protect (i) the proprietary weights of M and (ii) user-submitted feedback, which may contain sensitive user data and is subject privacy regulations. We additionally require S must not reveal which failure it is fixing when interacting with I, as it might inadvertently leak user data.
A Model Repair Institute I specialized in repairing text-to-image diffusion models. I wants to keep both its repairing algorithm and fix database secret, as they are its core intellectual property.
Goal and Technical Challenges. The above-mentioned failures make the world knowledge of M in deployments unaligned with users' values and expectations [15] 3 38 g Our goal is to repair M failures identified by U. Although users are essential for flagging failures, they do not directly participate in the repair process. Once the feedback is submitted, it becomes the responsibility of S to repair their model. As service providers usually lack expertise and resources for repairing failures (they mostly focus on enhancing image qualities), needs to contact an external Model Repair Institute I to perform such fixes. This is challenging for several reasons. First, service providers are not allowed to share user's s data") with hhrr--artiees due to privacy regulations. Second, service providers are not willing to hand over their models third parties due IP concerns. Third, Model Repair Institutes are not willing to disclose their fixes to service providers to protect their business model. Therefore,
two-party computation between S and I, with the feedback treated as private input held by S. We adopt the standard semi-honest security model (see Appendix D for the extension malicious security), where both parties follow the repair protocol correctly but may try to infer additional information from the interaction: both S and I are institutions with legal and reputational reasons to behave correctly during model repairing, though they may have incentives to recover more information.
model the protocol
Given the trust assumptions above, our goal is to build a provably secure protocol that protects the private inputs of both parties: the model weights and user feedback held by S, and the proprietary repair logic and database held by I. To achieve this, we rely on cryptographic techniques. We design a crypto-friendly knowledge editing algorithm by adapting an efficient editing method that avoids retraining from scratch. Based this, we construct a lightweight, customized two-party computation protocol, which we detail in the next section.
Our Solution.
4 SURE: SecUre model REpairing
We propose SURE, protocol for effective, efficient, and secure repair of text-to-image diffusion models based on user feedback and collaboration between a service provider S and model repair institute I. SURE combines crypto-friee friendly model repair algorithm with customized two-party computation (2PC) protocol. Our approach builds on recent knowledge editing techniques [30] that enable model updates without full retraining. However, applying these techniques out-of-the-box is unsuitable for efficient 2PC due to the large number of layers in diffusion models and the high cost of interactive operations such as high-dimensional matrix multiplications and inverses. Our key insight is that most of this cost can be avoided by carefully modifying the editing algorithm.
'Note that we do not consider protection of contidentiality or privacy of users' request to S as it is only failure identification and their institution knows their data, but want protect it against other institutions.

---

Algorithm 1: Repair database creation
Input: A Model Repair Institute I, A public text encoder (TextEncoder), A collection of Failures
Output: Repair Database
1: Repair Database = =
2: for all failure E Failures do
3: Repair data pair (source prompt:x, destination dessiaateestiinnoion
Creating repairing data {C R C' E R= TextEncoderi( [x, x'}) > Tokenizing and computing embeddings Creating embedding that corresponds to the same source token by discarding the embedding of additional tokens in the destination prompt
prompt:x'}
5: C' Text RemoveAddttionalTleenn
6: Wis iis (Ataiture C)C") (Aiuiture CC
Creating Repair Knowledge
Repair Database. append( (failure Wi,))
Output Repair Database
Algorithm Repair diffusion model parameters
Input: Service Provider S, A text-to-image diffusion model M, Received repair knowledge Wiix
Output: Updated parameters of the repaired text-to-image diffusion model M
I: CrossAttcntionnLayer + CroseAttentionAAces((()))) map textual data into visual data do
b Extract cross cross-attention layers that
2: for all i E Size(CrossAttentionlaayy
3: W Wi Wii. fix
Update value projection matrix Update key projection matrix
4: W <- W Wix
5: Updated diffusion model returned only the service provider
We design crypto-friendly repair algorithm (Section 4.1 tailored to the 2PC setting, without compromising the effectiveness of the original editing method. Our redesigned algorithm shifts almost all heavy computation offline, allowing each party to process its data locally and independently. Specifically: (1) I constructs the repair database offline (Algorithm [[] and (2) S applies the fix to its model parameters locally (Algorithm 2). In the online phase, further develop custom 2PC protocol (Section 4.2) that enables the service provider to securely locate and receive the fix corresponding to their failures from the institute's repair database through secure fuzzy matching procedure and a lightweight Oblivious Transfer (OT) protocol. We prove (Section 4.3 that our protocol keeps users' feedback, service provider's model parameters, and the institute proprietary editing algorithm confidential while ensuring that the model is faithfully repaired.
4.1
Crypto-Friendly Model Repair Algorithm
We instantiate SURE based on the Text-to-lmage Model Editing (TIME) procedure introduced by Orgad al. 330 We briefly review their core editing algorithm before presenting our modifications.
The editing algorithm in TIME takes as input two prompts:
source prompt, e.g., photo of CEO" that under-specifies certain visual attributes. It allows the model to fill in missing details using its implicit assumptions, which could reflect bias.
A more specific destination prompt, e.g., "a photo of female CEO" where explicit attribute is added to correct the failure in the original source prompt.
The editing goal to repair failures the model's original output by shifting the image generation from reflecting the source prompt to better align with the intended visual attributes of the destination prompt. This enables targeted correction of outdated, incorrect, or socially biased associations embedded in the model. The key insight from Orgad et al. is that it suffices update only the key and value projection matrices WK K and Wy (see Section for detailed definitions) within the model's 's cross-attens aattenttotnionnn layers. These matrices responsible for mapping textual tokens into intention-coppatiin visual representations, and patching them effectively alters the generated output.
Let {c:] C R and {c C R be the token embeddings of the source and destination prompt. For every source token, TIME locates the corresponding destination token that contains the same word and denotes its embedding by oz. This gives the aligned set {c] f for tokens appear both prompts. Let C [c1,...,CC and C c ..... cī]. for every layer the closed-form update
5

---

formula (Equation 5 in 30 is given by
W = -A (' + KKCT) CCT) " = (A + V''" (1)
where À E IR+ is a hyperparameter, and K* = = WxC* and = WyC
Efficiency and Privacy Challenges. The most direct way to securely evaluate the above update formula is to encode it as a circuit and run a generic 2PC: the provider S supplies the private weights W the institute I supplies C, C A, and the circuit outputs the updated weights W to S. Despite significant advances in modern 2PC protocols 43 201, applying them directly to this task remains inefficient. To illustrate the limitations of this generic approach, implemented baseline that computes the editing algorithm in a generic 2PC protocol, and it requires over 100 hours to perform single repair (see Section 5). More importantly because generic 2PC assumes the circuit is public, service provider will always learn the institute's ' proprictary repair algorithm.
Our Crypto-Friendly Editing Formula. The bottleneck above mainly comes from forcing heavy matrix operations into the secure computation. Our key observation that we can refactor the editing formula in way that completely eliminates any matrix operations inside 2PC. The matrix update formula in Equation I can be refactored as follow:
WV (A Wi + V'C") (ALL+
(Awi wir (AI+
w (AI (AI+C'")(AI-CC))
(2)
known S
Wt.. known to I
Here, S holds Wv, and I holds C, C*, and the hyperparameter A. Thus, I can compute
Wixx (AI+ (AI+CC")(++CC))
and S can update the matrix by computing W +- -W W.Wicc The above refactored equation applies identically to W k and holds for all layers in the model. The fix matrix Wix fix now encapsulates the semantics of the update and fully decouples model-specific parameters from repairs. Our refactored formula yields three immediate advantages for our purposes:
One fix fits all. The same Wix matrix can be reused across every cross-attention layer i, and applies uniformly to both W and W This significantly simplifies the repair process and reduces communication.
Matrix algebra disappears from 2PC. All matrix operations to compute Wix fi handled entirely by I offline. Then, S can use lighter cryptographic primitives like OT to acquire the Wix from I.
Algorithm privacy preserved. Because the fix is provided as single matrix and applied independently by S. there is no need to reveal the full structure of the editing algorithm or encode it into shared circuit. Therefore, I's proprictary repair method remains hidden from S.
We adopt this new editing formula in our protocol SURE. As we show in Section this seemingly simple refactor achieve four orders of magnitude speed up over the baseline. Appendix C describes the class of editing algorithms that our framework supports without incurring any utility loss.
4.2 Efficient Two-Party Model Repair Protocol
We now provide a detailed description of the secure two-party model repair protocol in SURE. The ideal functionality and our two-party model repair protocol are presented Figure Figure
We first briefly recall our setting. The protocol SURE involves two parties: a service provider S and a model repair institute I. S wish to repair its deployed model M and derives from aggregated user feedback a query key kayy E R* that captures the failure domain to be fixed. I maintains a private of size n. where each key k, E g* semantically labels
key-value repair database {k, Wix failure and each Wik.j the repair matrix for this failure. The protocol consists of three stages:
1. Database Initialization. Before interacting with S, the institute I locally computes Wixj from the embedding matrices C,, Cj and edit hyperparameter Ay. and tag the fix with a key k, that semantically describe the failure.

---

Functionality FRepair
This functionality is parameterized by similarity metric d(:,)) .) and database size 71.
Input:
S inputs a query key Kary E R and model matrices {Wi E ieee W E 1x Jie(m]-
Z inputs the database {k,, Ci, CT Ae hEef)) where C,, C, Rext A Et, and k, E IR*,
Model Repair:
1. Compute = arg minkets d(kery, k:), breaking ties by choosing the smallest i.
For each model layer [m], compute send the following updated matrices and index p S:
W
(A:W: N.W:+W:;,C)) (Aple C,,))) * (V,K).
Figure 2: Ideal functionality of model repair between S and I.
Protocol Ressirr
Input:
The service provider S and institute I agree similarity metric d(:,-) and the database size
S imputs fix query vector kary E R& and model matrices (Wi E Rexe, Wik k E R ]e(m). where m is the total number of model layers.
I inputs (k, C, Cj X5 bien), where C,, Cj E Rexx A3 E H. ky ie*, se*,a and n is the size of repair database.
Database Initialization: computes repair database (k: Wailbee where
Matching: Let Ca,, be the circuit that outputs(p, L), where p = arg minje(m) d(kgy ky), breaking tics by choosing the smallest j. S and I send (Ca, Kary) and (Ca,ns (kI,..., , kn)) ) to F2pc. 8 receives the fix matrix index p.
Model Repair:
and I send (recv, n, p) and (send, n. (Wexl)ee(n to For- S obtains matrix W Wixpp-
2. For each model layer i [m] and E {K, V.,, S locally updates each layer of its model using the same matrix: W + W. fix.p-
Figure 3: Our secure model repair protocol in the (For, FFrc) model.
2. Matching, S and Z run a small circuit inside 2PC to locate the database entry whose key k, minimizes a public similarity metric d(kary. k,). After this stage, only S learns the index p; I learns nothing about kgy beyond the fact that comparison occurred. When an exact match is sufficient-c.g, is the discrete metric or the database indexes are public. I can determine p outright, so this stage can be skipped and the parties proceed directly to the next step.
Oblivious Model Repair. After acquiring the index p. I runs an OT protocol to retrieve the single matrix Wix, Wix,p without revealing p and without accessing any other entry. It then updates every cross-attention layer locally by gggt-mmulippyyng both value and key projections with W fxxp to complete the repair.
Security Guarantees. Our protocol ensures that (i) the institute I learns nothing about the model M the query key kary: (ii) the service provider S learns only the single fix matrix matching its query and gains no information about any other entry in I's database; and (iii) the editing algorithm itself remains private, because I builds the database offline, the editing algorithm chosen by I remains entirely hidden from S. In the next section, we formalize and prove these guarantees.
4.3 Security Proof
In this section, establish the security of our protocol IR (Figure 3) and show how it can be generalized to any editing mechanism while hiding the editing algorithm being employed by the institute. All of our proofs are based the standard composition paradigm [5). We now state the following main security theorem of our protocol.
Theorem (Protocol Security). Protocol TRepair (Figure securely realizes FRepair (Figure in the (FoT, F2Pc) model against semi-honest adversaries.
7

---

Proof. For clarity. we denote the service provider by P1 and the repair institute by P2 for the remainder of the proof.
Correctness. Note that all matrix products in both the protocol and the functionality are well-defined. Additionally, for all [n], the regularization parameter >3 0, hence the matrix (AyI+ +C 0 and is therefore invertible.
To prove privacy, we separately consider the case of corrupted institute and service provider.
Corrupted Institute Pz. It is straightforward to prove security against P2, as it receives no output from either For and F2pc- Therefore, simulator S₂ that simply forwards P3 s message to FRepair can perfectly simulate its view.
Corrupted Service Provider P1. We construct simulator S that calls P1 as subroutine and interacts with RRepair to simulate its view. S1 proceeds as follows:
1. S1 obtains the message (Ca.n; kary) from Pi and record kery-
2. Sj sends (Kary: {Ic: [c}) to FRepair and receives index p and (WV, Waeim-
S₁ acts as F2pC and send to P1; upon obtaining (recv, n. p) from P,, send W to P.
We show that P1 's view is perfectly simulated. To see this, notice that the ideal world, because S₁ sends identity matrices to FRepair. for every layer
W = (ApLe + +, (111A,++CCCO +C,C,)) Wrippp
As a result, the matrix P1 received in the ideal execution is exactly the same from For in the real execution. Therefore, its view is perfectly simulated. As honest P3 receives output in both worlds, the joint output distributions are also identical in both worlds. This concludes the proof. L
Algorithm Privacy. For concreteness, instantiate protocol based the editing algorithm of [30]. However, our cryptographic construction readily accommodates any repair mechanism: any repair procedure that modifies model weights while leaving the network architecture unchanged can be dropped in without altering the protocol. Moreover, the protocol keeps the institute's choice of editing algorithm confidential. To see this, notice that S only sees the resulting fix matrix Wfx while the algorithm itself remains hidden. To formalize this property, we first define the notion of editing algorithms and prove theorem stating the algorithm-hiding property of our protocol.
Definition (Editing Algorithms). A model repair editing algorithm is an efficient mapping
f: (C, C", aux) W
where C, C' are the source and target prompt embedding matrices, aux is institute-hele auxiliary input, and Wix fix is the fix matrix of proper dimensions that is night-multppied to every model layer. Theorem (Algorithm Privacy). Let U, .......,Je fz} } be any finite family of editing algorithums. Let Repair be the extension of IRReaair in which chooses an index & E [Z], and builds its database using ft. Let Feepair be the corresponding ideal functionality that receives the description of fz from I. evaluates fz internally to obtain the fix matrices, and sends those matrices to S. Then, for every PPT adversary Ag corrupting the service provider S, there exists PPT simulator S₁ such that Ideal ( view viewReal(T) (TT Repair) view (FRepair).
Proof. The proof follows from the security proof of Theorem in straightforward manner. We define a modified simulator S that behaves the same as S1 in the security proof, except forward S's message to Repaiir instead of FRepair- It then follows from the security proof that IT Repair securely realizes Repaiir in the (FoT, (FFcc)--yybrr model and S perfectly simulates Ag's view.
Theorem 2 implies that the service provider S learns information about the institute's chosen editing algorithm fi beyond what is already implied by the fix matrix Wix.p- Consequently, our protocol enables the institute swap in or fine-tune its proprietary repair procedures without touching the underlying cryptographic protocol. This design not only supports fast iteration but also hides the institute's is editing knowledge from the service provider.
By contrast, generic secure 2PC protocols would require compiling the entire editing algorithm into a single Boolean or arithmetic circuit that is known to both parties-a standard assumption in secure
8

---

Functionality For
Upon receiving (send, n, (mi)ie(m) from Pt and (recv, n. from the Ps where b [ri], send my Pa.
Figure 4: Ideal functionality of I-out-of-n oblivious transfer.
Functionality Fapc
For E (1,2), upon receiving i.) from P., compute 91. C(x1, w2)) and send ye P.-
Figure 5: Ideal functionality of secure two-party computation.
computation constructions 46 16. In such settings, the circuit's structure (and hence the algorithm
encodes) is public, even if the inputs remain hidden.
5 Experimental Evaluation
We implement https://gitubb.oo//Geeettt-iiiii
and evaluate the efficiency of SURE in repairing Stable Diffusion v1.4 [34] with 32 layers, and compare it with a baseline model repair protocol that runs entirely within generic 2PC framework for comparison.
Experiment Setup. We implement our end-to-end protocol SURE and compare it with a baseline protocol that executes all editing operations within a generic 2PC framework. The baseline is implemented based on the semi protocol variant from the MP- SPDZ framework (BSD3 License) [22), popular framework for benchmarking generic secure protocols. However, MP-SPDZ does not provide a low-level OT interface suitable for our customized protocol in SURE. For ease of integration, we instead implemented SURE using the EMP-OT library from the EMP-toolkit (MIT License) [42),
which provides efficient implementations of various OT primitives and a flexible low-level API. To evaluate both SURE and the baseline, we perform single model repair on Stable Diffusion v1.4 (34) In this model, the source and target prompt embeddings C, C* are matrices of shape [768, 77]
The repair algorithm modifies 32 cross- -attsnnat layers in total, where each layer contains key and value projection matrices of shape [320, 768]. As a result, each fix matrix Wix has dimension [768, 768). We set the query and database keys kgry. (ke he[n represent all values using single-precision floating-poin numbers and use Euclidean distance as the similarity metric for key matching. All experiments are run with a single thread on two Amazon EC-2 c7i 2xlarge instances, each with 16 GB of RAM.
(00-dimensional vectors. We
Baseline. To highlight the efficiency of our lightweight protocol SURE, we implement baseline model repair protocol that runs entirely within generic 2PC framework for comparison. To ensure fair comparison, we apply several optimizations to avoid penalizing the baseline unnecessarily. First, we represent editing computation as an arithmetic circuit, which is more efficient than Boolean circuits for linear algebra. Additionally, we 32-bit fixed-point representation to avoid the high cost of floating-poit point arithmetic in 2PC. To reduce overhead further, we allow the model repair institute
I pre-compute the inverse matrices W (ApI. CPC outside the 2PC to avoid costly secure matrix inversion. The baseline protocol proceeds as follows:
1. S inputs the query key Kary and all projection matrices (Wi, k. W Helmm in all m layers; I inputs the repair database (k,, Ai C₁ C W Ieenl-
2. The circuit matches the closest index p = arg mingeti d(kgry k.) and breaks ties by choosing the smallest Then, for each layer i [mm] it computes the fix for all matrices:
w (p W + W.C.C w inv
* * (K,V),
3. The updated matrices (W K W heemm are then revealed to S.
Despite these optimizations, the baseline remains orders of magnitude slower than SURE.
Efficiency. F shows the end-to-end runtime and communication cost of SURE when perform- ing full Stable Diffusion v1.4 repair across varying repair database sizes. SURE completes the repair in under 17 seconds, even with repair database of 3,000 entries, with communication capped at 17.1 GB. Therefore, SURE is highly efficient.

---

Matching Model Repair
16.8s
Matching
17.IGB
15.2s
Communication (GB)
15.
Model Repair
14.3GB
Time (seconds)
11.1s
10
8.66
10
8.6GB
5.66
5,7GB
2.8s
2,9GB
500
1000
1,500
2000
2500
3000
500
1000
1500
2000
2500
3000
Database Size
Database Size
Figure 6: Runtime and communication cost for SURE to repair Stable Diffusion v1.4 with varying repair database sizes. Reported times are averages over 10 runs. Each database entry consists of a [768,768] fix matrix and a [100,1] key, where all numbers of the two parties' data sent. Costs are decomposed into two stages: (I) Matching finding the closest key database to failure query using Euclidean distance as similarity metric and (2) Model Repair returning the fix via the oblivious transfer protocol to S.
ponle-preecison floating-poii point. Communication reported is the larger
Table 1: Runtime and communication costs of SURE vs. the baseline approach to perform one repair of Stable Diffusion v1.4 with varying database sizes. SURE orders of magnitude faster than the baseline, our protocol completely avoids matrix operations secure computation.
Baseline
Ours
Database Size
Running Time (hours)
Comm. (TB)
Running Time (seconds)
Comm. (GB)
Running Time Improvement
500
167.36
76.42
2.81
2.94
2.14x105
1000
171.84
82.30
5.58
5.65
1.11x10s
1500
176.02
88.19
8.62
8.61
7.35x104
2000
180.48
94.08
11.10
11.41
5.8
2500
184.96
99.97
15.21
14.32
4.38x104
3000
189.44
105.86
16.77
17.09
x.07
Benchmarking. We further break down the total cost into main stages-key matching and model repair-as described in our protocol in Figure 3 Most of the runtime spent the matching stage, which uses lightweight 2PC protocol to identify the nearest key. In contrast, the OT-based model repair phase is highly efficient and remains nearly constant regardless of database sizes.
Scalability. Our protocol scales well with both the repair database and model size: since only single fix matrix is retrieved and applied across all layers, the online runtime independent of the number of model layers. Moreover, SURE's modular design allows for further optimization: the matching step can be replaced with more efficient cryptographic primitives such as private information retrieval fuzzy private set intersection. In cases where the database key is public the matching phase can be skipped entirely to further reduce overhead.
protocol against the baseline. Our protocol achieves up to 2 x 10 speedup. This dramatic improvement stems from avoiding expensive matrix operations and linear scans within 2PC. In the baseline, most of the cost arises from executing the entire editing formula securely and retrieving the correct matrix through full sean of the database, both of which scale poorly with the database size. In contrast, our customized design isolates the secure computation to a small matching task and a lightweight OT-based retrieval protocol, while offloading all matrix operations to local (offline) computation, resulting in far superior performance.
Comparison. Table T
compares
Acknowledgments
Work of Gefei Tan and Xiao Wang is partially supported by NSF awards #2318975 and #2236819.
30 considers a database gender bias in different professions, where the database key is simply the profession name (e.g.. "nurse"), which is made public. Because the index of the desired fix is known in advance, matching is unnecessary.
For example,
10

---

References
[1] Adobe Inc. Adobe firefly. https: htttps:/www... adobe. wtw.:adbbb..c.o.oooooo..o. httppe//..oooooooccc////re////// html 2025-05-14.
Accessed
[2] Artbreeder. Inc.
Artbreeder.
htttps://addailyyrop com/
artbreeder-an -at--aa-----ttee-e -at--rrrtgneetorr// Founded by Joel Simon, Accessed 2025-05-14.
[3] Samyadeep Basu, Nanxuan Zhao, Vlad I Morariu, Soheil Feizi, and Varun Manjunatha. Local- izing and editing knowledge in tex--to-image generative models. In The Twelfth International Conference on Learning Representations, 2023.
[4] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee. Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/dallle-3 pdf. 2(3):8. 2023.
[5] Ran Canetti, Security and composition of multiparty cryptographic protocols. Journal of Cryptology, 13(1):143-200 January 2000.
[6] Canva Pty Ltd. Canva magic media (dream lab). https://en. http://eawwwiippiaa....a org/wikt/ http/wiik/Caavaa Accessed 2025-05-14.
[7] Pu Cao, Feng Zhou, Qing Song, and Lu Yang. Controllable generation with text-to-image diffusion models: A survey. arXiv preprint arXiv:2403.04279 2024.
[8] Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5253-5270, 2023.
[9] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307. 15217, 2023.
[10] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.
[11] Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A. Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35(1):53-65 2018.
[12] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image syn- thesis. In Marc Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 8780-878 8794, 2021.
[13] Rohit Gandikota, Joanna Materzyúska. Jaden from diffusion models. In 2023 IEEF/CVF International Conference on Computer Vision (ICCV), pages 2426-2436, 2023.
and David Bau. Erasing concepts
[14] Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyúska, and David Bau. Unified concept editing in diffusion models. arXiv preprint arXiv:2308. 2023.
[15] Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyíska, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5111-5120, 2024.
[16] Oded Goldreich, Silvio Micali, and Avi Wigderson. How play any mental game A completeness theorem for protocols with honest majority. In Alfred Aho, editor, 19th ACM STOC, pages 218-229. ACM Press, May 1987.
[17] Google LLC.
Google imagefx.
https://blog htts://bloggooglle/beeee woggle/ecchoooogg///ooooooo/////
google- labs- imagefx imagefx-textfx foxtfx-gonerativer al/ Via Google Labs/ / Google DeepMind, Accessed 2025-05-14.
[18] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis, 2022.
11

---

[19] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc.
[20] Zhicong Huang, Cheng Hong, Chenkai Weng, Wen-jie Lu, and Hunter Qu. More efficient secure matrix multiplication for unbalanced recommender systems. IEEE Transactions on Dependable and Secure Computing, 20(1):551-562) 2023.
[21] Ideogram. Inc. Idcogram 3.0. https://en. wikipedia. wrapiiaorrrwiiii//eegorr.e
(text- imdiieemeemee) Accessed 2025-05-14.
[22] Marcel Keller. MP-SPI7:: A versatile framework for multi-party computation. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, 2020.
[23] Yeongmin Kim, Byeonghu Na, Minsang Park, JoonHo Jang, Dongjun Kim, Wanmo Kang, and Il-Chul Moon. Training unbiased diffusion models from biased dataset. CoRR, abs/2403.01 1889, 2024.
[24] Leonardo Interactive Pty Ltd. Leonardo ai. https: aubius/eessbb.sinsssss.. .wweesss.oo...o gu/ www.au/ABN/Viee. ABN/View? id-56662209485) Trading as leonardo.ai, Accessed 2025-05-14.
[25] Microsoft Corp. Bing image creator and microsoft designer. https://wwx. http//wwwwbbnn....m com/
create Powered by OpenAl, Accessed 2025-05-14.
[26]
Midjourney, Inc. Midjourney. httpp:: htttsss://e.iiiiiddii.... httppeiia.org/iilii/i/oooorraa Accessed 2025-05-14.
[27] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022.
[28] NightCafe Studio Pty Ltd. Nightcafe studio. https://nightcer http://iigttcae...succc//////pss about-nightcafe, Accessed 2025-05-14.
[29]
OpenAI, Inc. Dall-e 3. https:
wwtom/indx/ddaat daap////eddddd////// Accessed 2025-05-14. 14.
[30]
Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions in text- to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 7053-7061, 2023.
[31] Yong-Hyun Park, Sangdoo Yun, Jin-Hwa Kim, Junho Kim, Geonhui Jang, Yonghyun Jeong, Junghyo Jo. and Gayoung Lee. Direct unlearning optimization for robust and safe text-to-image models, arXiv preprint arXiv:2407.21033 2024.
[32] Michael O. Rabin. How exchange secrets with oblivious transfer. 2005. https://eprint
iacr. iacr. orcc.orr/2000/18
[33] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents, 2022.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[35]
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, 2023.
[36]
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Lit, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Raphael Gontijo-Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text- to-image diffusion models with deep language understanding. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc.
[37] Xudong Shen, Chao Du, Tianyu Pang, Min Lin, Yongkang Wong, and Mohan S. Kankanhalli. Finetuning text-to-image diffusion models for fairesss. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 1, 2024. OpenReview.net, 2024.
[38] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffu- sion art or digital forgery? investigating data replication diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6048-6058, 2023.
12

---

[39] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Curran Associates Inc., Red Hook, NY, USA, 2019.
[40] Stability AI Ltd. Stable diffusion dreamstudio. https:/ httpe://rraassudio.sfaaiiiiy aii/ Accessed 2025-05-14.
[41] European Union. Regulation (eu) 2016/679 (general data protection regulation).
https
//eur-lex europa http:x.uurpp...eeeeeeeeeee htt/ell/reee//////e//e/ 2791/6796/// 2016.
[42] Xiao Wang, Alex J. Malozemofff and Jonathan Katz. EMP-toolkit Efficient multiparty compu- tation toolkit. httpp://githhb. com/ emp- toolkit] 2016.
[43] Xiao Wang, Samuel Ranellucci, and Jonathan Katz. Authenticated garbling and efficient mali- ciously secure two-party computation. In Bhavani M. Thuraisingham, David Evans, Tal Malkin, and Dongyan Xu, editors, ACM CCS 2017, pages 21-37. ACM Press, October / November 2017.
[44] Tianwei Xiong, Yue Wu, Enze Xie, Yue Wu, Zhenguo Li, and Xihui Liu. Editing massive concepts in text-to-image diffusion models, 2024.
[45] Pengfei Yang, Ngai-Man Cheung, and Xinda Ma. Text to image generation and editing: A survey, arXiv preprint arXiv:2505.0235
2025.
[46] Andrew Chi-Chih Yao. How generate and exchange secrets. In 27th Annual Symposium on Foundations of Computer Science (sfcs 1986), pages 162-167, 1986.
13

---

Appendices
A
Limitations
SURE the first secure framework for model editing and demonstrates high efficiency even on commercial-scee text-to-image diffusion models. However, it is currently limited this domain. Extending to other models, such as LLMs, may require new editing algorithms and cryptographic techniques. Also, efficient protocol only supports linear editing of the model weight; if editing techniques involve architectural changes or non-linear weight updates, further optimizations might need be made to maintain its efficiency. While SURE can handle multiple repairs, optimization, batching, or amortization is implemented. We comment that the matching phase can be further optimized for multiple repair settings using techniques like fuzzy private intersection private information retrieval.
B Ideal Functionalities
The ideal functionality of 1-out-of-n OT is depicted in Figure This can be efficiently realized using log n l-out-of-2 OT, which can in turn be efficiently computed using existing cryptographic protocols. The ideal functionality of 2PC is presented in Figure
C On the Utility and Scope of Supported Editing Algorithms of SURE
In this section, clarify the relationship between cryptographic protocol SURE and the underly- ing model editing algorithms is designed to protect. We detail the preservation of editing utility and discuss the class of algorithms framework supports.
C.1 SURE Preserves the Utility of the Underlying Editing Algorithm
central point of our work is that SURE cryptographic protocol, not new model editing algorithm. Its purpose is execute an existing editing algorithm in a secure, privacy-preservee manner using two-party computation (2PC). Our protocol does not alter the underlying mathematical operations of the editing algorithm itself. Additionally, in our implementation, all numerical values floating points, which ensures that the numerical
are represented using standard accuracy is identical to the original, non-private computation. Consequently, the utility, efficacy, and potential side effects of an edit performed using SURE are identical to those of the original algorithm (e-g., TIME (3) UCE [14).. All quantitative and qualitative evaluations from the original papers-such as editing effectiveness, generalization, concept specificity, and impact on general image quality (FID/CLIP scores) are directly applicable to edits performed with our method.
C.2 SURE Supports More Advanced Algorithms
Our protocol is designed to efficiently support a general class of editing algorithms: any algorithm that can be expressed as linear transformation of the model's weights. As formalized in the main body of our paper, this class includes any editing algorithm where the update to a layer's weight matrix W can be refactored into matrix multiplication with "fix matrix" Wix-
Crucially, our framework supports the more powerful UCE algorithm 14 without incurring any additional computational overhead. UCE is capable of performing complex batch edits, such as debiasing multiple attributes or erasing up to 100 artistic styles simultaneously. Its closed-form update rule (equation 7 in the paper) is given as:
w'
Σ J.c GEE
Σ Wolde;e
(E c.O
Σ -gee))
cGEP
CEEE
where E and P are sets of editing target and preserving target and v² Wolds We can refactor the formula similar way by plugging in o*:
w' = Wold Σ
ce +
Σ c
(E qo + Σ Ege
GEE
GEP
GEE
GEP
W &s
14

---

where Wir fix is the terms in brackets. Notice that, like TIME, computing the Wiix is independent of W ola. and therefore can prepared by the repair institute before any interaction.
While TIME and UCE apply a single, global fix matrix to all edited layers, our protocol is not restricted to this paradigm. Our framework specific fix matrices with only negligible additional overhead. In this scenario, the dominating key matching phase of our protocol remains constant, while only the Oblivious Transfer (OT) cost increases minimally with the number of distinct matrices.
easily and naturally generalize to support layer-
D Extending SURE to Malicious Security
We comment that our protocol's s modular design allows for direct extension to the malicious security setting. The overall protocol structure will largely remain the same and consists of two minor changes: (1) replacing the semi-honest cryptographic primitives with their maliciously secure counterparts and (2) adding lightweight consistency check to ensure the potentially malicious Service Provider will provide the same index it retrieved from the matching phase to the OT functionality. Because the matching circuit and number of oblivious transfers both very small, switching to their malicious version will not blow up the overall runtime. Additionally, the consistency check has a constant cost independent of the database size.
We estimate the running time of malicious secure SURE using malicious OT and 2PC subprotocol, and implement the malicious version of the baseline using the maliciously secure protocol variantmascot from the MP-SPDZ. Table shows the runtime breakdown both our semi-honest and malicious protocols compared to the baseline implementation. Maliciously secure SURE increases the total runtime by roughly 9x compared to the semi-honest one. The overhead almost entirely incurred by the Matching phase, which runs in malicious 2PC. In contrast, the Model Repair phase, which only requires few oblivious transfers, incurs very little additional cost. Malicious security overhead can be avoided in scenarios where the matching phase is not needed. In such cases, the protocol only needs few malicious OTs and can skip the expensive 2PC and the consistency check; the cost of malicious security introduced by Ollog n) malicious OT becomes negligible.
Table 2: Runtime of SURE with semi-honest and malicious security vs. the baseline approach to perform one repair Stable Diffusion v1.4 with varying database sizes. Maliciously secure SURE increases the total runtime by roughly 9x compared to the semi-honest version.
Baseline
Ours
Database Size
Improvement
Malicious
Malicious
Semi-honest
Total (h)
Repair (s)
Total (s)
Repair (s)
Total (s)
500
2860.91
0.55
23.35
0.54
2.82
4.41
10
1000
2937.49
1.08
46.61
1.05
5,60
2.27
10
1500
3008.95
1.58
72.19
1.53
8.60
1.50 x 10a
2000
3085,95
2.11
92.53
2.05
11.09
1.20 x 105
2500
3161.77
2.66
129.19
2.59
15.24
8.81
10a
3000
3238.35
3.17
140.53
3.09
16.83
8.30 X 104
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our evaluation presented in Section validatess our efficiency claims. Security proof' in Section 4.3 validate our security claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide a separate limitation section in Appendix A
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

We detail our threat model and trust assumptions in Section 3 and provide standard security proofs for all our claims in Section 4.3
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: For reproducibility of our algorithms, our core algorithms described in Sec- contain all details and can be independently implemented using most existing crypto- graphic libraries. For reproducibility of our results, we report our experiment setup details in Section 5 including the exact environments and implementation frameworks. Our code for all experiments can be found https://github..om//efee Gefee--Te/eiTRRRB
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We include all data/codebase required to reproduce all of our experimental results in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips. //nips.cc/pub.lee wwtpc//pppp.guudeee.ooeppppp htwpl///guuddss/odeeuubbiissses irnooeSubmissionPol111.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 5 we detailed our experiment environment and setups. Our code for all experiments can be found at https://gtthbb.co//ffffeTTaa TRRRSE
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our experiments evaluate the runtime and communication cost of the proposed cryptographic protocol. For each reported data point (each bar in Figure (6). we ran the protocol over 10 independent executions and report the average to capture system-level variability. We clarify this in Figure 6 that our running time is the average of 10 repetitions of our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We detailed our experiment testbed in Section 3 and the running time reported for each experiment (Figure (6) can be used to estimate the time of execution for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'tttt make it into the paper).
paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics https: pttttp:://neurpptt http://wwrrppssccccpllllol///e// httpp////p//tttecccccceee....aa.
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification: Authors have carefully read the Code of Ethics and confirm that the research conducted conforms
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
19

---

Justification: Throughout the paper (e.g., in Section 1 and 3) we discuss the positive societal impact of our work as a preservinn technique that enables model repair without exposing user feedback, proprietary model parameters, proprietary repair algorithms. This helps foster responsible deployment of generative models by allowing stakeholders to address biased or outdated behavior while respecting privacy constraints.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not have high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The two open source frameworks we used for our implementation are properly credited and their licenses and terms are properly followed.
Guidelines:
20

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide a well-documented codebase for our protocol implementation with the supplemental materials.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
21

---

Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification:
This paper does not describe the usage of LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy /ttp://oooriipcoo/ooooorrrr///// for what should or should not be described.
22


Reviews:
[{"review_id": "WyldyDyPCN", "reviewer": "Reviewer_zFKo", "summary": "This paper studies the confidential collaboration problem between a diffusion model service provider and a model repair institute. When a diffusion model service user submitted unexpected outputs to the service provider, it's important for the provider to identify and fix such misalignment. While it’s common for a service provider to seek collaboration from external third-party model repair institutes, it’s important to keep the confidentiality of both the provider's proprietary model and user feedback, as well as the repair institutes’ proprietary repair technique. This paper proposes such a secure repairing method, SURE, that encodes the prior Text-to-Image Model Editing (TIME) method into a secure two-party computation (2PC) protocol. The key insight is to identify a decomposition of the computation in TIME to factor out the heavy matrix-multiplication from the 2PC protocol. This paper provides security proof for the proposed method. This paper experimentally evaluated it against a vanilla TIME implementation in 2PC without optimizations, and the results shows that the proposed (optimized) method can significantly reduce the runtime and communication overhead in practice.", "questions": "- Regarding the efficiency, what are the assumptions on the editing algorithm to fit this framework to reduce the overhead?\n\n- Regarding the functionality, can this framework support more general repair requests, like align a prompt with a (batch of) images, and general repair methods, like a fine-tuning based method?\n\n- How can another editing method be plugged-in this framework? What are the requirements?\n\n- Can the protocol be extended so that the service provider does not lose access to prior repairs even if the model repair institutes stopped the service? For example, allowing the service provider to store the database and do the database matching offline?\n\n- Minor presentation suggestions: \n\n  - Regarding Table 1, it's recommended to place the caption on the top of a table.\n\n  - If possible, it might be more intuitive to align the unit of time and data for the baseline and SURE in Table 1 to highlight how significant the improvement is. For example, the time unit can be inlined like “167.36h” vs. “2.81s”. This is just a suggestion but not a demand.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The problem is well-motivated. The problem of efficient and confidential collaboration between service providers and model repair institutes is interesting.\n\n- The idea of incorporating existing editing algorithms with the 2PC protocol to achieve confidential collaboration is novel to this community.\n\n- This work identified an effective optimization in encoding the TIME editing algorithm in a 2PC protocol, which significantly reduced the computation and communication overhead and made the protocol practical.\n\n## Weaknesses\n\n- The proposed framework lacks discussion of assumptions/requirements for a general editing algorithm to fit and gain the overhead reduction. Without such discussion, the proposed method is better to be viewed as an optimized 2PC extension to the TIME algorithm, but not as a general framework.", "was_revised": true, "final_justification": "The revision plan proposed by the authors has addressed my concerns.\n\nAlthough being limited to a narrow subset of \"crypto-friendly\" editing methods, I agree that the proposed idea is novel and solid, and can have practical impact for those applicable architectures and methods. I'll keep my rating as borderline accept.", "rebuttal": "We thank you for noting the strengths of our paper: \n- interesting and well-motivated problems; \n- novel and practical solutions. \n\nWe respond below to all questions raised.   \n### **What are the assumptions on the editing algorithm to fit this framework to reduce the overhead?**\n### **Can this framework support more general repair requests, like align a prompt with a (batch of) images, and general repair methods, like a fine-tuning based method? How can another editing method be plugged-in this framework? What are the requirements?**\n\n**TL;DR: Our protocol assumes the editing can be expressed as a linear transformation of the model’s weight. These editing methods can all be made crypto-friendly through our optimisation. Our protocol directly supports advanced, state-of-the-art editing algorithms like UCE [Ref.1], which can perform batch alignment.**\n\nIn theory, it is feasible to execute any editing algorithm securely in 2PC. However, as we show in our baseline, even for TIME, directly plugging it into a 2PC protocol incurs prohibitive costs. To make this practically feasible and mitigate the substantial computational overheads inherent to both 2PC and knowledge editing algorithms, we show that optimisations on both machine learning and cryptography fronts are essential: i) designing a crypto-friendly editing algorithm; ii) developing a customised secure protocol tailored to it.  \n\nIn this paper, we instantiate one such category of crypto-friendly algorithms using a closed-form repair formula that satisfies two key properties: \nEnabling expensive operations to be performed offline, entirely eliminating matrix operations within the 2PC protocol; \nBeing agnostic to the number of layers, allowing the same fix to be reused across all layers.  \n\nAs formalized in Definition 1, this class includes any editing algorithm that can be refactored as applying a fixed linear transformation to a layer. In particular, we refactor TIME editing algorithm (state-of-the-art in knowledge editing as it is highly efficient and offers a closed-form solution) such that its repair modifies the weight matrix by right-multiplication it with a “fix matrix”: $W’ = WW_{fix}$. \n\nWithout incurring any additional costs, our framework can support multiple edits at the same time using \"Unified Concept Editing in Diffusion Models\" (UCE) [Ref 1]. UCE is capable of repairing multiple targets in a single edit: it can erase up to 100 artists simultaneously before significantly impacting image fidelity or CLIP scores on unrelated concepts. UCE has a more complex closed-form solution that is capable of repairing multiple targets in a single edit, which contains a set for desired fixes and a specific “preservation term” to further prevent undesirable editing artifacts. We observed that the closed-form formula of UCE can also be refactored, and the fix can be encapsulated into a fix matrix $W_{fix}$ of the same size. Therefore, our protocol can directly support it without additional overhead.\n\nSpecifically, UCE gives the following closed-form update formula (equation 7 in the paper):\n$W =\\Bigl(\\sum_{c_i\\in E} v_i^* c_i^\\top+\\sum_{c_j\\in P} W_{{old}} c_j c_j^\\top\\Bigr)\\Bigl(\\sum_{c_i\\in E} c_i c_i^\\top+\\sum_{c_j\\in P} c_j c_j^\\top\\Bigr)^{-1}$\nwhere $E$ and $P$ are sets of editing target and preserving target and $v^* = W_{old}c_i^* $.\nWe can refactor the formula in a similar way by plugging in $v*$\n$W = W_{{old}}\\Bigl(\\sum_{c_i\\in E} c_{i}^* c_i^\\top+\\sum_{c_j\\in P} c_j c_j^\\top\\Bigr)\\Bigl(\\sum_{c_i\\in E} c_i c_i^\\top+\\sum_{c_j\\in P} c_j c_j^\\top\\Bigr)^{-1}$ , \nwhere $W_{fix}$ is the terms in brackets.\nNotice that, like TIME, computing the $W_{fix}$ is independent of $W_{old}$, and therefore can be prepared by the repair institute before any interaction. \n\nWhile both TIME and UCE algorithms use a single fix matrix for all edited layers, our protocol easily and naturally generalises to support layer-specific fix matrices with only negligible additional cost. \nSpecifically, the Oblivious Transfer (OT) cost will slightly increase with the number of matrices, but the dominating key matching phase will remain constant. This means the total overhead remains low even when handling multiple distinct fix matrices.\n\nWe will summarize and incorporate this clarification into the paper accordingly. \n\n### **Can the protocol be extended so that the service provider does not lose access to prior repairs even if the model repair institutes stopped the service? For example, allowing the service provider to store the database and do the database matching offline?**\nThat is an excellent point regarding the long-term availability of repairs. Our protocol inherently supports this need, although through a different mechanism than the one suggested.\nWhile the repair database itself is the intellectual property of the repair institute and cannot be shared (see Section 3, line 141), our framework offers a more direct solution. Once the service provider securely obtains the fix through our protocol, they can store it locally and retain it indefinitely. This stored fix can then be reapplied to the current model at any time. Moreover, it can even be applied across multiple different future models that require the same correction, without any further interaction with the repair institute. Therefore, even if the repair institute were to stop its operations, the service provider would not lose access to any previously acquired repairs. This is in contrast to the generic baseline, which requires a costly interactive 2PC execution for every single model repair, even if the fix has been previously applied and the model is the same.\nWe appreciate the insight and have highlighted this important property in our problem description.\n\n### **Minor presentation suggestions**\nThank you for your careful reading and helpful suggestions regarding the presentation of our paper. We have incorporated both presentation suggestions.\n\n\n[Ref 1] Gandikota, R., Orgad, H., Belinkov, Y., Materzyńska, J., & Bau, D. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024."}, {"review_id": "G0nE6G1nDb", "reviewer": "Reviewer_QmFt", "summary": "The paper proposes SURE, which is a light-weight two-party protocol (2PC) that lets a model owner obtain a weight-patch matrix to fix a user-flagged error in diffusion model while keeping weights, user feedback, and the editing method hidden. In this protocol, the repair institute precomputes a fix matrix which the service provider can multiply it to all cross-attention layers locally, which enables a full model repair with small traffic and the protocol is proven secure in the semi-honest model.", "questions": "- Have you run any quantitative or qualitative studies to show that a patched model actually corrects the user-flagged error without degrading other outputs?\n- How would the protocol/runtime/cost have to change to change to a maliciously secure setting?\n- Is there a possibility of a stronger choice of baseline other than a full linear-scan MPC?\n- Is it possible to apply several fixes to one prompt at a time? Does the cost increase in this case?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n- The paper tackles a well-motivated problem that is relevant to real-world service providers of diffusion models, as it aims to fix errors in systems that are already deployed while complying to security constraints.\n- The paper provides tight security guarantees, with proofs in the hybrid model as well as their algorithm-hiding property of the protocol. \n- The method results in a substantially more efficient end-to-end repair efficiency compared to a baseline method with ~17s running time for a 3,000 sized database. \n\n**Weakness**\n- The paper does not discuss malicious-security guarantees or expected change of cost (compared to semi-honest setup) but this kind of setting might be required for a real-world use case.\n- The paper discusses only Stable Diffusion v1.4, which has a significant gap between more recent generation models that are typically deployed and might need repairs in a real-world setup. Runtime analysis for larger models such as SDXL or SD3 might be necessary to demonstrate generalizability.\n- The baseline might be too weak, which might make the comparison a bit unrealistic.", "was_revised": true, "final_justification": "The paper provides a solid method and justification but is limited to the old diffusion model architectures, which somewhat limit my excitements about this work, points being similar to the one raised by reviewer xN9B. Overall, I believe this work provides a solid solution to a niche problem, but is limited in some aspects, therefore I keep my score of weak accept.", "rebuttal": "Thank you for noting the strengths of our paper: \n- well-motivated and real-world problem; \n- end-to-end efficient solutions with tight security guarantees. \n\nWe address your questions below.\n\n### **The paper does not discuss malicious-security guarantees or expected change of cost (compared to semi-honest setup) but this kind of setting might be required for a real-world use case. How would the protocol/runtime/cost have to change to change to a maliciously secure setting?**\nOur protocol’s modular design allows for a direct extension to the malicious security setting. The overall protocol structure will largely remain the same and consists of two minor changes: (1) replacing the semi-honest cryptographic primitives with their maliciously secure counterparts and (2) adding a lightweight consistency check to ensure the potentially malicious Service Provider will provide the same index it retrieved from the matching phase to the OT functionality.  Because the matching circuit and number of oblivious transfers are both very small, switching to their malicious version will not blow up the overall runtime. Additionally, the consistency check has a constant cost independent of the database size.\n\nWe estimate the running time of malicious secure SURE using Malicious OT and 2PC subprotocol. We also report the malicious version of the baseline (using the mascot protocol variant from the MP-SPDZ). Maliciously secure SURE increases the total runtime by ~9x compared to our semi-honest implementation. Note that the overhead is almost entirely incurred by the Matching phase, which runs in a malicious 2PC. In contrast, the Model Repair phase, which only requires a few oblivious transfers, incurs very little additional cost. We comment that malicious security overhead can be avoided in scenarios where the Matching phase is not needed (as we discussed at Line 329, footnote 3). In such cases, the protocol only needs a few malicious OTs and can skip the expensive 2PC and the consistency check; the cost of malicious security introduced by O(logn) malicious OT becomes negligible.\n\n Database Size | **Repair Only (Malicious)** | Repair Only (Semi-Honest) | **Matching+Repair  (Malicious)** | Matching+Repair (Semi-Honest) | **Matching+Repair (Baseline,Malicious)** \n---|---|---|---|---|---\n 500 | 0.55s | 0.54s | 23.35s | 2.82s | 2860.91h \n 1000 | 1.08s | 1.05s | 46.61s | 5.60s | 2937.49h \n 1500 | 1.58s | 1.53s | 72.19s | 8.60s | 3008.95h \n 2000 | 2.11s | 2.05s | 92.53s | 11.09s | 3085.95h \n 2500 | 2.66s | 2.59s | 129.19s | 15.24s | 3161.77h \n 3000 | 3.17s | 3.09s | 140.53s | 16.83s | 3238.35h \n\n\n### **Stable Diffusion v1.4 has a significant gap between more recent generation models that are typically deployed and might need repairs in a real-world setup. Runtime analysis for larger models such as SDXL or SD3 might be necessary to demonstrate generalizability.**\nWe note that our work, SURE, is the first privacy-preserving protocol to support complex, real-world diffusion models. We view identifying repair algorithms that can be made crypto-friendly as one of our main contributions, as it's rare for cryptographic protocols to scale to non-trivial, real-world machine learning models of this size without resorting to approximations that would degrade performance. Our protocol achieves this efficiently, performing repairs in **seconds**.\nWe chose Stable Diffusion v1.4 specifically because it is a widely-used, large-scale model that serves as a common benchmark in both academic research and deployed products, and it is the benchmark model used in the foundational editing literature (TIME, UCE) that our protocol implements. This allows us to directly validate that SURE applies edits with zero loss of utility compared to the original, non-private algorithms.\nWhile Stable Diffusion v1.4 serves as our validation target, our protocol is designed to scale efficiently to even larger models. Its scalability can be analyzed by two factors: \n\n**Models with more layers:** A key advantage of our protocol is that its cost is independent of the number of layers being edited. The fix matrix $W_{fix}$ is retrieved once through the interactive protocol, and the service provider can then apply this same fix to all relevant layers of the model locally, without any further interaction. In contrast, the baseline edits each layer with a separate execution of a secure 2PC protocol for matrix operations, which makes the total cost scale linearly.\n\n**Models with larger weight matrices:** Larger models could have higher-dimensional weight matrices, which increase the size of the corresponding $W_{fix}$ . In our protocol, this only impacts the communication cost. The total number of Oblivious Transfer (OT) operations remains the same, but the size of the payload within each OT message increases linearly with the size of the $W_{fix}$.\n\n### **Have you run any quantitative or qualitative studies to show that a patched model actually corrects the user-flagged error without degrading other outputs?**\n\nThe unique feature of TIME is that it does not change the underlying Model editing logic/operation at all. Although we did clear refactoring to the editing algorithm, it only makes it more cryptography-friendly while maintaining the identical effect. Therefore, our method provides the exact same utility as the original algorithm. The quantitative and qualitative studies demonstrating that TIME successfully corrects errors without significant degradation to other outputs can be found in the original paper and its follow-up work.\n\n\n### **Is there a possibility of a stronger choice of baseline other than a full linear-scan MPC?**\nAs we discussed in Appendix C (line 828), our baseline already includes significant optimizations for matrix operations to be a fair point of comparison. Specifically, we allow the repair institute to locally pre-compute parts of the repair formula. This strategy eliminates the most expensive matrix inverse operations and reduces the number of matrix multiplications inside the 2PC protocol. Despite these optimizations, the secure matrix computation remains the overwhelming bottleneck: for a database of 500 entries, the secure edit takes around 167 hours while the linear scan only takes ~500 seconds. This shows that even if we were to use a more customized protocol like Private Information Retrieval to make the matching phase faster, the baseline would remain impractical. \n\n\n### **Is it possible to apply several fixes to one prompt at a time? Does the cost increase in this case?**\nThanks for the great question. Yes, our protocol directly supports multiple fixes through a single protocol execution, and the core cryptographic cost does not increase. The only potential cost increase relates to the size of the repair database, which is a manageable trade-off.\n\nOur protocol is compatible with follow-up work on TIME, “Unified Concept Editing in Diffusion Models” [Ref 1], which generalizes the closed-form solution to incorporate multiple editing targets into a single update. Their formula can also be refactored using our optimisations as $W’ = W_{old} W_{fix}$ in a similar way, which makes it compatible with our protocol. Specifically, UCE gives the following closed-form update formula (equation 7 in the paper):\n$W =\\Bigl(\\sum_{c_i\\in E} v_i^* c_i^\\top+\\sum_{c_j\\in P} W_{{old}} c_j c_j^\\top\\Bigr)\\Bigl(\\sum_{c_i\\in E} c_i c_i^\\top+\\sum_{c_j\\in P} c_j c_j^\\top\\Bigr)^{-1}$\nwhere $E$ and $P$ are sets of editing target and preserving target and $v^* = W_{old}c_i^* $.\nWe can refactor the formula in a similar way by plugging in $v*$\n$W = W_{{old}}\\Bigl(\\sum_{c_i\\in E} c_{i}^* c_i^\\top+\\sum_{c_j\\in P} c_j c_j^\\top\\Bigr)\\Bigl(\\sum_{c_i\\in E} c_i c_i^\\top+\\sum_{c_j\\in P} c_j c_j^\\top\\Bigr)^{-1}$ , \nwhere $W_{fix}$ is the terms in brackets.\nNotice that, like TIME, computing the $W_{fix}$ is independent of $W_{old}$, and therefore can be prepared by the repair institute before any interaction. \n\nWhen applying UCE using our protocol, each $W_{fix}$ encodes multiple editing targets. The cost can be analyzed in two parts:\nThe core cryptographic cost of our protocol involves securely retrieving one $W_{fix}$ matrix. Since the multi-edit $W_{fix}$ ​has the same size as a single-edit one, the computation and communication for one run of the protocol do not change at all.\nIncreased database size: The potential cost increase comes from the growth of the repair database. To support combined edits, the database must store pre-computed entries for those combinations. For example, a simple database might have entries for fixing gender bias for “nurse”, “engineer”, and “cook”. But to support multiple edits, the database would need to store all possible combinations of these three professions. However, our protocol is modular to make this manageable. For extremely large databases, the default matching component can be swapped with advanced solutions like Private Information Retrieval (PIR), which reduces the matching cost to be sub-linear in the database size.\n\n\n[Ref 1] Gandikota, R., Orgad, H., Belinkov, Y., Materzyńska, J., & Bau, D. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024."}, {"review_id": "12LmGLHszb", "reviewer": "Reviewer_b8Yo", "summary": "This paper introduces SURE, a novel end-to-end framework capable of securely and blindly repairing post-deployment failures, such as inconsistent image generations caused by outdated, incorrect, or biased assumptions. SURE adopts a co-design approach, optimizing machine learning and cryptographic components to facilitate confidential collaboration between service providers and model repair institutes. By refactoring the editing algorithm into a \"crypto-friendly\" form and utilizing a customized two-party computation (2PC) protocol combined with oblivious transfer (OT), SURE avoids expensive matrix operations in secure computation, offloading a significant amount of computation offline. Experiments demonstrate remarkable practical efficiency, achieving a four-order-of-magnitude improvement over a generic 2PC baseline. This work successfully demonstrates the feasibility of practical and secure model repair for large-scale diffusion services, ensuring continuous alignment with evolving societal expectations without compromising sensitive data or proprietary knowledge.", "questions": "1.Questionable Generality of W_fix: How generalizable is the \"One fix fits all\" claim for W_fix across diverse repair tasks beyond simple bias correction (e.g., object removal, style alteration), and what experimental evidence supports its non-destructive application to such varied failures?\n2.Unevaluated Impact on Unrelated Generation Tasks: What is the impact of modifying all 32 cross-attention layers on the model's general generative capabilities for unrelated prompts, and do these global modifications introduce unintended artifacts or reduce quality for unaddressed tasks?\n3.Lack of In-depth Qualitative Evaluation and Repair Examples: Could the paper provide detailed qualitative evaluations, including before-and-after image comparisons, to demonstrate the visual quality, semantic consistency, and lack of new artifacts in repaired images, as quantitative metrics alone are insufficient?\n4.Insufficient Verification of Failure Type Diversity: How does SURE's effectiveness generalize to a broader range of failure types beyond bias correction, such as outdated information, factual inaccuracies, or stylistic changes, and what experimental evidence supports this versatility?\n5.Scalability and Maintenance Costs of the Repair Database: What are the practical implications and costs associated with scaling and maintaining the \"repair database,\" particularly concerning its construction, efficient retrieval, and conflict resolution when multiple W_fix entries apply or conflict?\n6.\"Blindfolded Repair\" Context and Limitations: Given the title \"Blindfolded Repair,\" could the paper more rigorously define the scope of privacy and the adversarial models covered, especially concerning malicious attackers or sensitive PII in user feedback?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strengths：\n1. This paper innovatively introduces the SURE framework, offering the first end-to-end solution for addressing \"failures\" (such as biases or outdated knowledge) that inevitably arise in large-scale diffusion models during real-world deployment. Crucially, it achieves this while preserving the confidentiality of user feedback, the Service Provider's proprietary model, and the Model Repair Institute's specialized knowledge. This is a groundbreaking problem with significant practical implications and academic value in the current landscape of widespread deep learning model application.\n2.The SURE framework achieves secure repair of diffusion models through the co-design of a model editing algorithm with customized Two-Party Computation (2PC) protocols. This \"co-design\" approach is a core technical contribution of the paper, enabling extremely high repair efficiency while maintaining privacy. Experimental results demonstrate that SURE achieves orders-of-magnitude speed improvements compared to generic baselines, proving the practicality and efficiency of its lightweight protocol.\n3.SURE implements model repair by identifying and modifying specific parameters, namely the key and value projection matrices (W_fix) within the diffusion model's cross-attention layers. This strategy, which targets specific parameters, effectively localizes the repair operation to the critical parts influencing model behavior, demonstrating a deep understanding and sophisticated application of the internal mechanisms of diffusion models.\n\nWeaknesses:\n\n1.Questionable Generality of W_fix: The paper claims \"One fix fits all,\" implying that the same W_fix matrix can be reused across all cross-attention layers. However, this \"one-shot\" assertion requires more detailed explanation and rigorous validation. The paper primarily uses the specific bias repair of \"CEO generating female CEO\" as an example but does not sufficiently explore or experimentally demonstrate whether a W_fix generated for one specific type of error (e.g., removing a certain object) can be effectively and non-destructively applied to other vastly different repair tasks (e.g., altering image style or correcting factual inaccuracies). Different types of \"failures\" may necessitate different kinds of knowledge modifications, and whether a single W_fix structure can handle all cases efficiently and precisely remains an unproven claim. If the generality of W_fix is limited only to repairs within specific semantic categories, then the claim of \"one fix fits all\" appears overly broad.\n2. SURE repairs problems by modifying all 32 cross-attention layers of the diffusion model. Such global modifications could have side effects on the model's image generation capabilities for unrelated or non-repair-related text prompts. The paper fails to adequately assess the potential negative impact of the repair operation on the model's overall generation quality, diversity, and unaffected generation tasks. For instance, after fixing the \"CEO gender bias,\" would the model introduce new artifacts or suffer reduced quality when generating \"landscape photos\" or \"animal pictures\"? This is a common challenge for all model editing methods, and the paper should provide a more detailed analysis and experimental results.\n3.Lack of In-depth Qualitative Evaluation and Repair Examples. Despite claiming that SURE is \"highly practical\" and offers significant speed improvements, the paper lacks detailed qualitative evaluation of the repaired images. It does not provide comparative examples of images before and after repair, nor does it discuss in detail the visual quality, semantic consistency, or the introduction of new artifacts in the repaired images. Relying solely on quantitative metrics (such as speed) is insufficient to fully demonstrate the effectiveness of the repair method. Providing actual examples of \"female CEO\" images generated after fixing the \"CEO gender bias,\" along with comparisons to the original problematic images, would greatly strengthen the argument's persuasiveness.\n4. The paper primarily discusses model bias, such as gender bias. However, the scope of \"failures\" can be very broad, including outdated information, factual inaccuracies, removal or addition of specific styles, etc. The paper does not adequately demonstrate SURE's ability to handle and repair diverse types of \"failures.\" For more complex or nuanced repair tasks, SURE's effectiveness remains unknown.\n5.The paper mentions that the Model Repair Institute (I) needs to maintain a \"repair database.\" However, there is no in-depth discussion regarding the practical scale of this database, its construction and maintenance costs, or how to ensure efficient retrieval and repair when the number of database entries becomes vast. As the types of model failures continuously increase, the scalability of the database and its associated computational and storage overhead could become a bottleneck for practical deployment. Furthermore, if multiple potential W_fix entries are applicable to the same query, or if different W_fix entries conflict, how does SURE handle these situations?\n6.The title \"Blindfolded Repair\" implies complete privacy for all parties involved. However, this \"blindfolded\" aspect may not hold true in certain real-world adversarial scenarios, such as when a malicious attacker actively deviates from the protocol to gain information, or in situations where user feedback itself contains highly sensitive Personally Identifiable Information (PII). The paper should more rigorously define the scope of privacy covered by \"blindfolded repair\" and the adversarial models it addresses.", "was_revised": true, "final_justification": "After reading the rebuttals and other reviewers' comments, I think this is a good paper but there are still some drawbacks to be solved. Thus I kept my score as  Borderline Accept.", "rebuttal": "Thank you for your review and valuable feedback! \nWe want to clarify a crucial point that addresses your primary concerns (Questions 1-4). Our work introduces SURE, a novel, privacy-preserving protocol for applying model edits, which does not alter the underlying mathematical editing algorithm. Therefore, the utility and effectiveness of an edit applied via SURE are identical to those of the original algorithm. We will add a discussion to clarify this in our manuscript.\nTo address your concern regarding the effectiveness of TIME's “one fix fits all” nature and demonstrate how our flexible protocol can benefit from future algorithmic improvements, we consider the follow-up work Unified Concept Editing (UCE) [Ref. 1]. UCE generalizes TIME's closed-form formula to handle multiple, distinct fixes in a single update.  Our protocol fully supports this multi-target editing without loss of efficiency, as the resulting $W_{fix}$ matrix prepared by the repair institute is the same size. Specifically, UCE gives the following closed-form update formula (equation 7 in the paper):\n$W =\\Bigl(\\sum_{c_i\\in E} v_i^* c_i^\\top+\\sum_{c_j\\in P} W_{{old}} c_j c_j^\\top\\Bigr)\\Bigl(\\sum_{c_i\\in E} c_i c_i^\\top+\\sum_{c_j\\in P} c_j c_j^\\top\\Bigr)^{-1}$\nwhere $E$ and $P$ are sets of editing target and preserving target and $v^* = W_{old}c_i^* $.\nWe can refactor the formula in a similar way by plugging in $v*$\n$W = W_{{old}}\\Bigl(\\sum_{c_i\\in E} c_{i}^* c_i^\\top+\\sum_{c_j\\in P} c_j c_j^\\top\\Bigr)\\Bigl(\\sum_{c_i\\in E} c_i c_i^\\top+\\sum_{c_j\\in P} c_j c_j^\\top\\Bigr)^{-1}$ , \nwhere $W_{fix}$ is the terms in brackets.\nNotice that, like TIME, computing the $W_{fix}$ is independent of $W_{old}$, and therefore can be prepared by the repair institute before any interaction. \n\nGiven that our protocol faithfully applies these algorithms, the extensive evaluations from the TIME and UCE papers directly answer your questions (1-4) about the editing method's performance. Both papers provide extensive quantitative and qualitative evaluations for (1) efficacy, which measures how effective the editing method is on the editing prompt, (2) generality, which measures how the editing method generalizes to other related prompts, and (3) specificity, which measures the ability to leave the generation of unrelated prompts unaffected. We summarize these results below.\n\n[Ref 1] Gandikota, R., Orgad, H., Belinkov, Y., Materzyńska, J., & Bau, D. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024.\n### **Q1: Generality of one fix fits all.**\nBoth TIME and UCE show extensively that this closed-form formula is very effective for a wide range of repair tasks (including object removal and style alteration). For example, it could \n- modify object attributes: (e.g., changing a rose's color, a pizza's shape)\n- completely remove concepts (removing a \"garbage truck\" from the model's knowledge) \n- apply style alteration (erase artistic styles (removing the influence of a specific artist) \n- handle complex, multi-attribute debiasing.\n- perform combinations of these edits simultaneously without a loss in effectiveness.\n\n### **Q2: Impact on Unrelated Generation Tasks**\nModifying all layers does not harm unrelated images. The editing is highly targeted to only a small fraction of the model's parameters (2.2% for Stable Diffusion 1.4). In both TIME and UCE, evaluations confirmed that:\n1. Modifying specific concepts does not affect others. For example, editing \"roses\" to be blue does not change the colour of \"poppies\".\n2. TIME does not degrade the overall generative quality of the model after editing (wrt FID and CLIP Score)\n3. UCE achieves high fidelity and minimal distortion to generations for unaddressed tasks, even after extensive batch edits.\n4. UCE’s editing formula contains a specific “preservation term” to further prevent undesirable editing artifacts. \n\n### **Q3: Lack of In-depth Qualitative Evaluation/Repair Examples**\nThe TIME and UCE paper presents extensive before-and-after image comparisons demonstrating its effectiveness, generalizability, which includes changing object attributes (\"a pack of roses\" to \"blue roses\"), context (\"a cow\" to \"a cow on the beach\"), and correcting biases. They also demonstrate that edits generalize logically to related prompts (e.g., editing \"house\" also affects \"an oil painting of a house\"). They also provide visual examples that confirm that unrelated prompts remain pristine (e.g., \"poppies\" are unchanged when \"roses\" are edited). We are happy to add them to our paper if you think it is useful. \n\n### **Q4: Insufficient Verification of Failure Type Diversity**\nThe effectiveness and generalisation abilities of TIME and UCE directly apply to SURE. Their evaluations and visual examples show that the editing generalises very well beyond bias correction while having little undesirable artifacts on unaffected prompts.\n1. Outdated information: TIME fixes outdated information, such as when a celebrity changed their hairstyle. They edit the Harry Potter Franchise character \"Hagrid\" to become \"Blond Hagrid\". It shows that it generalizes well for prompts containing the concept of Hagrid (e.g. “a painting of Hagrid”) and doesn't affect other Harry Potter characters. \n2. Factual Inaccuracies: Both TIME and UCE can override models’ inaccurate \"implicit assumptions\" about object attributes and align them with desired factual alterations, including altering colors, shape, materials, and completely erasing the concept of an object.\n3. Stylistic Changes: TIME can induce stylistic shifts by modifying conceptual attributes (e.g., changing the material of a house). UCE can remove specific artistic styles, and experiment shows it can erase up to 100 artists simultaneously before significantly impacting image fidelity or CLIP scores on unrelated concepts.\n\n### **Q5: costs with scaling repair database: construction, efficient retrieval, and when multiple $W_{fix}$ entries apply or conflict?**\nRegarding scalability, we note that the offline costs for constructing the database are generally negligible compared to the cryptographic online cost, as they only involve standard plaintext matrix operations, which are inexpensive and highly parallelizable on modern hardware.\n\nAs for the online cost, the database size will not affect the efficiency of retrieval much (model repair phase in Figure 4). The retrieval only requires $\\mathcal{O}(\\log n)$ number of oblivious transfers: concretely, for 1 billion entries, it will only require 30 oblivious transfers, although the final running time will depend on the bandwidth, as communication unavoidably will scale linearly with the database size. Database size will mainly affect the cost of the matching phase, but our protocol is modular to handle this: for practical, domain-specific databases of moderate size, a simple linear scan is concretely efficient. For massive-scale databases, the matching component can be swapped with advanced sub-protocols like Private Information Retrieval to achieve sub-linear performance that can easily handle millions of records.\n\nFor conflict resolution, we break ties by entry index, mainly for technical consistency. In practice, for a well-curated database with a precise similarity metric, it is unlikely that equally similar entries have significant conflicts in fixes. Furthermore, if a conflicting or low-quality fix were applied, the service provider could readily detect the resulting model degradation through local benchmarking.\n\n### **Q6: \"Blindfolded Repair\" Context and Limitations: the scope of privacy and adversarial models, especially concerning malicious attackers or sensitive PII in user feedback.**\nThe title \"Blindfolded Repair\" refers to the privacy guarantee of our protocol: the repair institute provides fixes without learning the model weights or which specific fix a service provider is requesting, and the service provider obtains a needed fix without learning anything else about the repair database or the underlying repairing algorithm.\n\nOur implementation is under the semi-honest threat model (line 122) and we specified our assumption for each party in Section 3. Regarding sensitive PII, we assume the user voluntarily hands their feedback to the service provider, but the service provider must not disclose this to any third party (in practice, this is reasonable under privacy regulations). \nFor security against malicious adversaries, our protocol can be easily extended to a malicious setting by (1) replacing cryptographic primitives (2PC and OT) with their malicious secure counterparts and (2) a lightweight consistency check between the matching output and the OT input to ensure a  Service Provider correctly uses the index from the Matching phase in the subsequent steps. We estimate the cost of a maliciously secure version of SURE and compare it with a maliciously secure baseline (implemented using the MASCOT protocol from MP-SPDZ). We summarize the running time in the table below.\n\n\n Database Size | **Repair Only (Malicious)** | Repair Only (Semi-Honest) | **Matching+Repair  (Malicious)** | Matching+Repair (Semi-Honest) | **Matching+Repair (Baseline,Malicious)** \n---|---|---|---|---|---\n 500 | 0.55s | 0.54s | 23.35s | 2.82s | 2860.91h \n 1000 | 1.08s | 1.05s | 46.61s | 5.60s | 2937.49h \n 1500 | 1.58s | 1.53s | 72.19s | 8.60s | 3008.95h \n 2000 | 2.11s | 2.05s | 92.53s | 11.09s | 3085.95h \n 2500 | 2.66s | 2.59s | 129.19s | 15.24s | 3161.77h \n 3000 | 3.17s | 3.09s | 140.53s | 16.83s | 3238.35h \n\nMalicious security increases the total runtime by ~9x. This overhead comes almost entirely from the Matching phase executed in malicious 2PC. However, in applications where the Matching phase is not required (as noted in footnote 3), our protocol bypasses the costly 2PC. In this case, the consistency check can also be avoided, and the $\\mathcal{O}(\\log n)$ numbers of malicious OT adds virtually no overhead."}, {"review_id": "sPUl2OVSwG", "reviewer": "Reviewer_xN9B", "summary": "This paper proposes a novel privacy-preserving knowledge-sharing framework for knowledge editing in text-to-image models, addressing the scenario involving a model developer and a repair institute. In this context, the institute aims to conceal proprietary repair techniques, while the developer seeks to avoid disclosing model weights. To address these privacy constraints, the authors introduce a privacy-preserving model repair methodology that leverages Oblivious Transfer (OT) as a core mechanism to safeguard the confidentiality of both parties. However, the direct application of OT suffers prohibitively high computational overhead due to its inherent complexity. To mitigate this limitation, the framework optimizes the process by decomposing existing model repair methodologies (TIME) into modular components, thereby significantly reducing computational costs while preserving the privacy guarantees of OT.", "questions": "1. could you further elaborate why service providers can not be the a model repair institute? In other word, what is the main challenge of the model repair task?\n\n2. Can this method be applied to erasing method that does not have a closed-form solution? Can this method be applied to methods with very complex closed-form solution?\n\n3. What is the effectiveness of the proposed method? In detail, is the encrypted editing method (TIME) still works? More qualitative and quantitative results should be provided.\n______________\nOverall, I find the proposed scenario practical and interesting. However, I'm still not sure whether it is a **real problem** in the world (question 1). Besides, I'm not sure if more complex editing methods can be encrypted (question 2 and 3). I will raise my score if these concerns are addressed.", "limitations": "The authors have already stated the limitations of their proposed method.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n+ The proposed framework demonstrates significant novelty and practical orientation, addressing a pressing issue that closely mirrors real-world challenges in model deployment.\n\n+ The method to disentangle the update weight in concept erasing method (TIME) is clever and it can help to improve the efficiency of the privacy preserving method.\n\n## Weaknesses\n\n+ I find the paper not very friendly to those who are not familiar with cryptography. I can follow each step of the proposed method. However, I still do not understand how privacy is preserved. I suggest adding more preliminaries about this to facilitate those who are not familiar with cryptography. \n\n+ The proposed method’s reliance on closed-form solutions for model editing restricts its applicability. LoRA-based methods or approaches requiring iterative weight updates (e.g., fine-tuning) cannot be directly supported. Even for techniques with closed-form solutions, decomposition into lightweight and precomputed components may fail if the closed-form equations are inherently complex. This narrow scope limits the framework’s potential for broader adoption in diverse editing scenarios.\n\n+ Though the scenario described in the intro is very practical, I still find several defaults. First, why service providers can not repair their models? (I'm not convinced by the reasons given in line 35-38). Model repairing (or editing, erasing) is not a complex task (at least in this paper, TIME is not a complex method). Given this, why service providers can not repair their models?\n\n+ The assumption that model service provider lacks the ability to perform repairs requires stronger justification. Given that concept-erasure methods like TIME are quite straightforward (e.g., matrix operations), the assumption that model service provider would outsource such tasks to third parties seems under-supported. The authors should provide empirical evidence (e.g., case studies) or domain-specific constraints to validate this scenario.\n\n+ typos: line 2 test $\\rightarrow$ text", "was_revised": true, "final_justification": "I lean towards acceptance as this paper proposes an interesting while very practical concerns in the real world. However, I’m hesitant give a higher score as the problem formulation is not very rigorous and the provided solution is somewhat narrow.", "rebuttal": "We thank you for noting the strengths of our paper, namely: \n- Significant novelty and practical orientations\n- Addressing a pressing problem with real-world challenges\n- Clever optimizations \n\nWe fixed all typos and respond below to all questions raised. \n\n### **paper is not very friendly to those who are not familiar with cryptography**\nThank you for your suggestions! We will add more crypto preliminaries in the text. Here's an intuitive summary of how privacy is preserved and the related concepts.\n\n**Secure Multi-Party Computation (MPC):** MPC enables multiple parties to jointly compute an agreed-upon function of their inputs. The protocol guarantees that no information beyond the output is revealed. 2PC (Two-Party Computation) is the specialized case for two participants.\nWe use a 2PC protocol for the Matching Phase. The 2PC matching function takes the provider's matching key (semantically, “fix gender bias in nurse”) and the institute's set of database keys as inputs. It only reveals the index of the correct match to the service provider. The index is later used to retrieve the fix matrix via OT.\n\n**Oblivious Transfer (OT):** Oblivious Transfer is a protocol between a sender and a receiver. The sender has multiple messages, and the receiver wants to retrieve one of them. OT’s privacy guarantee ensures that (1) the sender learns nothing about which item the receiver chose, and (2) the receiver learns nothing about the items they didn't choose.\nOur protocol uses OT in the Model Repair phase. The service provider (receiver) privately retrieves the specific $W_{fix}$ matrix it needs from the repair institute's (sender's) database without revealing which one it picked.\n\n**Semi-Honest and Malicious Security:** A semi-honest (or honest-but-curious) adversary correctly follows the protocol specification but may attempt to infer additional information. A malicious adversary may arbitrarily deviate from the protocol. We implement and evaluate our protocol with semi-honest security, but our protocol can also be extended to malicious security without significant overhead. We estimated the maliciously secure version of our protocol and present the numbers in the table below.\n Database Size | **Repair Only (Malicious)** | Repair Only (Semi-Honest) | **Matching+Repair  (Malicious)** | Matching+Repair (Semi-Honest) | **Matching+Repair (Baseline,Malicious)** \n---|---|---|---|---|---\n 500 | 0.55s | 0.54s | 23.35s | 2.82s | 2860.91h \n 1000 | 1.08s | 1.05s | 46.61s | 5.60s | 2937.49h \n 1500 | 1.58s | 1.53s | 72.19s | 8.60s | 3008.95h \n 2000 | 2.11s | 2.05s | 92.53s | 11.09s | 3085.95h \n 2500 | 2.66s | 2.59s | 129.19s | 15.24s | 3161.77h \n 3000 | 3.17s | 3.09s | 140.53s | 16.83s | 3238.35h \n\n**How privacy is preserved:** Our protocol works in two main steps: (1) private matching via 2PC and (2) private fix retrieval via OT. The privacy is preserved directly through the security properties of 2PC and OT primitives. Our protocol is formally proven secure under the standard simulation paradigm. \nThe core idea is to first define a hypothetical, ideal-world trusted party that handles the entire model repair (Figure 2). By definition, this process is perfectly secure. Then, the proof shows that our real-world protocol “behaves the same” as the ideal-world trusted party. Since the ideal functionality is secure by definition, it follows that our protocol is just as secure and leaks no unwanted information.\n\n### **Why service providers can't be repair institute? What is the challenge of the model repair task?**\nThe primary challenge in model repair is not the computational cost of applying a fix but the accurate diagnosis of a failure from limited user feedback. We argue it is challenging for service providers to repair their model for several key reasons:\n1. **Insufficient Feedback for Diagnosis:** Modern generative models typically rely on simple feedback mechanisms like a \"thumbs up/down\" button[Ref. 2]. This single bit of information is ambiguous and lacks the context needed to identify a specific failure. Without extensive expertise of user reports from the repair institute, interpreting a \"thumbs down\" is guesswork.\n2. **Lack of Specialized Domain Knowledge:** Even with feedback, service providers cannot possess the deep domain expertise required to pinpoint subtle errors. They may not be familiar with the nuances of niche topics (e.g., the inaccurate details of a sports car), the evolving standards for evaluating social bias, or the specifics of a factual inaccuracy. They can't fix a problem they don't have the expertise to understand.\n3. **Inability to Keep Knowledge Current:** A model's implicit knowledge can become outdated overnight due to news, cultural events, or even a celebrity changing their hairstyle, or other significant events after its knowledge cutoff. It is not the core business of a service provider to track these real-world changes, making it difficult for them to identify and repair obsolete information.\n\nFor example, consider a user who prompts for an image of the \"latest iPhone\" shortly after a new model is released and gives a \"thumbs down\" to the generated image of an older model. A service provider, seeing a plausible modern smartphone, would be unable to diagnose the error from this vague feedback. They may lack the knowledge to distinguish the subtle design changes between models, and it is not their core business to track real-time product releases. In contrast, a specialised repair institute would immediately recognize that the model's knowledge is outdated and provide the correct fix.\n\n### **Can this method be applied to erasing method that does not have a closed-form solution? Can this method be applied to methods with very complex closed-form solution?**\nOur method cannot be directly applied to iterative methods, as our focus is on creating a protocol that is both crypto-friendly and highly efficient. While such methods could theoretically use generic secure computation, our baseline shows that even simple edits become prohibitively inefficient, making iterative approaches infeasible in practice.\nHowever, we note that our framework is highly versatile and supports advanced, state-of-the-art editing methods. Closed-form editing solutions handle erasing tasks very well, and our protocol supports highly complex closed-form solutions. The core principle is that as long as the final weight update can be expressed as $W’ = WW_{fix}$, our protocol can apply it, regardless of the complexity involved in deriving $W_{fix}$. The follow-up work on TIME titled \"Unified Concept Editing in Diffusion Models\" (UCE) [Ref 1] is the perfect example of this. UCE provides a more complex closed-form solution that is capable of repairing multiple targets in a single edit, which contains a set for desired fixes and a specific “preservation term” to further prevent undesirable editing artifacts. We observed that the closed-form formula of UCE can also be refactored, and the fix can be encapsulated into a fix matrix $W_{fix}$ of the same size. Therefore, our protocol can directly support it without additional overhead.\n\nSpecifically, UCE gives the following closed-form update formula (equation 7 in the paper):\n$W =\\Bigl(\\sum_{c_i\\in E} v_i^* c_i^\\top+\\sum_{c_j\\in P} W_{{old}} c_j c_j^\\top\\Bigr)\\Bigl(\\sum_{c_i\\in E} c_i c_i^\\top+\\sum_{c_j\\in P} c_j c_j^\\top\\Bigr)^{-1}$\nwhere $E$ and $P$ are sets of editing target and preserving target and $v^* = W_{old}c_i^* $.\nWe can refactor the formula in a similar way by plugging in $v*$\n$W = W_{{old}}\\Bigl(\\sum_{c_i\\in E} c_{i}^* c_i^\\top+\\sum_{c_j\\in P} c_j c_j^\\top\\Bigr)\\Bigl(\\sum_{c_i\\in E} c_i c_i^\\top+\\sum_{c_j\\in P} c_j c_j^\\top\\Bigr)^{-1}$ , \nwhere $W_{fix}$ is the terms in brackets.\nNotice that, like TIME, computing the $W_{fix}$ is independent of $W_{old}$, and therefore can be prepared by the repair institute before any interaction. \n\n### **Effectiveness of the proposed method: does the encrypted editing method (TIME) still works? Need qualitative and quantitative results**\nOur method is instantiated by TIME and can also support its more powerful follow-up UCE as discussed above. Crucially, our method does not change the underlying editing algorithm itself. In our experiment, all numbers are represented using standard single-precision floating point, so the numerical accuracies will be identical as well. Therefore, the utility and effectiveness of an edit applied via SURE are identical to those of the original algorithm, and the comprehensive evaluations of the original TIME and UCE paper can be directly adopted to conclude the effectiveness of SURE, which we briefly summarize below.\n\nQualitatively, the TIME algorithm is shown to work on a wide range of tasks. It can change object attributes like making roses blue, update factual information such as making a character blond, and apply various stylistic changes. Quantitatively, TIME demonstrates high efficacy in making the intended edit and strong generality in applying that change to related concepts. While there is a slight trade-off, it largely maintains specificity, leaving unrelated images untouched. Importantly, standard image quality metrics like FID and CLIP scores remain stable, which means that the overall model quality is preserved. Furthermore, our protocol supports the more advanced UCE algorithm, which extends these capabilities to complex tasks. UCE can perform multi-attribute debiasing for gender and race and is powerful enough to erase up to 100 artistic styles simultaneously without significant degradation to the model.\n\n[Ref 1] Gandikota, R., Orgad, H., Belinkov, Y., Materzyńska, J., & Bau, D. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024.\n\n[Ref. 2] Collins et al. (2024). Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation."}]

Author rebuttal:

Meta-review: The paper received unanimously positive scores from the reviewers. The cite well-motivated framework and problem, proofs, security guarantees, and novelty, to mention a few. There was a healthy discussion with reviewers, which further strengthen the importance of the paper. Hence the decision is to accept. Congrats!
Decision: Accept (poster)