Title: Pin the Tail on the Model: Blindfolded Repair of User-Flagged Failures in Text-to-Image Services
Venue: NeurIPS 2025
Accepted: yes
Keywords: Model Editing

PPML

Secure Computation

Diffusion Models

Cryptography
Abstract: Diffusion models are increasingly deployed in real-world text-to-image services. These models, however, encode implicit assumptions about the world based on web-scraped image-caption pairs used during training. Over time, such assumptions may become outdated, incorrect, or socially biased--leading to failures where the generated images misalign with users' expectations or evolving societal norms. Identifying and fixing such failures is challenging and, thus, a valuable asset for service providers, as failures often emerge post-deployment and demand specialized expertise and resources to resolve them. In this work, we introduce $\textit{SURE}$, the first end‑to‑end framework that $\textbf{S}$ec$\textbf{U}$rely $\textbf{RE}$pairs failures flagged by users of diffusion-based services. $\textit{SURE}$ enables the service provider to securely collaborate with an external third-party specialized in model repairing (i.e., Model Repair Institute) without compromising the confidentiality of user feedback, the service provider’s proprietary model, or the Model Repair Institute’s proprietary repairing knowledge. 
To achieve the best possible efficiency, we propose a co-design of a model editing algorithm with a customized two-party cryptographic protocol.
Our experiments show that $\textit{SURE}$ is highly practical: $\textit{SURE}$ securely and effectively repairs all 32 layers of {Stable Diffusion v1.4} in under 17 seconds (four orders of magnitude more efficient than a general baseline). Our results demonstrate that practical, secure model repair is attainable for large-scale, modern diffusion services.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel privacy-preserving knowledge-sharing framework for knowledge editing in text-to-image models, addressing the scenario involving a model developer and a repair institute. In this context, the institute aims to conceal proprietary repair techniques, while the developer seeks to avoid disclosing model weights. To address these privacy constraints, the authors introduce a privacy-preserving model repair methodology that leverages Oblivious Transfer (OT) as a core mechanism to safeguard the confidentiality of both parties. However, the direct application of OT suffers prohibitively high computational overhead due to its inherent complexity. To mitigate this limitation, the framework optimizes the process by decomposing existing model repair methodologies (TIME) into modular components, thereby significantly reducing computational costs while preserving the privacy guarantees of OT.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n+ The proposed framework demonstrates significant novelty and practical orientation, addressing a pressing issue that closely mirrors real-world challenges in model deployment.\n\n+ The method to disentangle the update weight in concept erasing method (TIME) is clever and it can help to improve the efficiency of the privacy preserving method.\n\n## Weaknesses\n\n+ I find the paper not very friendly to those who are not familiar with cryptography. I can follow each step of the proposed method. However, I still do not understand how privacy is preserved. I suggest adding more preliminaries about this to facilitate those who are not familiar with cryptography. \n\n+ The proposed method’s reliance on closed-form solutions for model editing restricts its applicability. LoRA-based methods or approaches requiring iterative weight updates (e.g., fine-tuning) cannot be directly supported. Even for techniques with closed-form solutions, decomposition into lightweight and precomputed components may fail if the closed-form equations are inherently complex. This narrow scope limits the framework’s potential for broader adoption in diverse editing scenarios.\n\n+ Though the scenario described in the intro is very practical, I still find several defaults. First, why service providers can not repair their models? (I'm not convinced by the reasons given in line 35-38). Model repairing (or editing, erasing) is not a complex task (at least in this paper, TIME is not a complex method). Given this, why service providers can not repair their models?\n\n+ The assumption that model service provider lacks the ability to perform repairs requires stronger justification. Given that concept-erasure methods like TIME are quite straightforward (e.g., matrix operations), the assumption that model service provider would outsource such tasks to third parties seems under-supported. The authors should provide empirical evidence (e.g., case studies) or domain-specific constraints to validate this scenario.\n\n+ typos: line 2 test $\\rightarrow$ text\n\nQUESTIONS:\n1. could you further elaborate why service providers can not be the a model repair institute? In other word, what is the main challenge of the model repair task?\n\n2. Can this method be applied to erasing method that does not have a closed-form solution? Can this method be applied to methods with very complex closed-form solution?\n\n3. What is the effectiveness of the proposed method? In detail, is the encrypted editing method (TIME) still works? More qualitative and quantitative results should be provided.\n______________\nOverall, I find the proposed scenario practical and interesting. However, I'm still not sure whether it is a **real problem** in the world (question 1). Besides, I'm not sure if more complex editing methods can be encrypted (question 2 and 3). I will raise my score if these concerns are addressed.\n\nLIMITATIONS:\nThe authors have already stated the limitations of their proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SURE, a novel end-to-end framework capable of securely and blindly repairing post-deployment failures, such as inconsistent image generations caused by outdated, incorrect, or biased assumptions. SURE adopts a co-design approach, optimizing machine learning and cryptographic components to facilitate confidential collaboration between service providers and model repair institutes. By refactoring the editing algorithm into a \"crypto-friendly\" form and utilizing a customized two-party computation (2PC) protocol combined with oblivious transfer (OT), SURE avoids expensive matrix operations in secure computation, offloading a significant amount of computation offline. Experiments demonstrate remarkable practical efficiency, achieving a four-order-of-magnitude improvement over a generic 2PC baseline. This work successfully demonstrates the feasibility of practical and secure model repair for large-scale diffusion services, ensuring continuous alignment with evolving societal expectations without compromising sensitive data or proprietary knowledge.\n\nSTRENGTHS AND WEAKNESSES:\nstrengths：\n1. This paper innovatively introduces the SURE framework, offering the first end-to-end solution for addressing \"failures\" (such as biases or outdated knowledge) that inevitably arise in large-scale diffusion models during real-world deployment. Crucially, it achieves this while preserving the confidentiality of user feedback, the Service Provider's proprietary model, and the Model Repair Institute's specialized knowledge. This is a groundbreaking problem with significant practical implications and academic value in the current landscape of widespread deep learning model application.\n2.The SURE framework achieves secure repair of diffusion models through the co-design of a model editing algorithm with customized Two-Party Computation (2PC) protocols. This \"co-design\" approach is a core technical contribution of the paper, enabling extremely high repair efficiency while maintaining privacy. Experimental results demonstrate that SURE achieves orders-of-magnitude speed improvements compared to generic baselines, proving the practicality and efficiency of its lightweight protocol.\n3.SURE implements model repair by identifying and modifying specific parameters, namely the key and value projection matrices (W_fix) within the diffusion model's cross-attention layers. This strategy, which targets specific parameters, effectively localizes the repair operation to the critical parts influencing model behavior, demonstrating a deep understanding and sophisticated application of the internal mechanisms of diffusion models.\n\nWeaknesses:\n\n1.Questionable Generality of W_fix: The paper claims \"One fix fits all,\" implying that the same W_fix matrix can be reused across all cross-attention layers. However, this \"one-shot\" assertion requires more detailed explanation and rigorous validation. The paper primarily uses the specific bias repair of \"CEO generating female CEO\" as an example but does not sufficiently explore or experimentally demonstrate whether a W_fix generated for one specific type of error (e.g., removing a certain object) can be effectively and non-destructively applied to other vastly different repair tasks (e.g., altering image style or correcting factual inaccuracies). Different types of \"failures\" may necessitate different kinds of knowledge modifications, and whether a single W_fix structure can handle all cases efficiently and precisely remains an unproven claim. If the generality of W_fix is limited only to repairs within specific semantic categories, then the claim of \"one fix fits all\" appears overly broad.\n2. SURE repairs problems by modifying all 32 cross-attention layers of the diffusion model. Such global modifications could have side effects on the model's image generation capabilities for unrelated or non-repair-related text prompts. The paper fails to adequately assess the potential negative impact of the repair operation on the model's overall generation quality, diversity, and unaffected generation tasks. For instance, after fixing the \"CEO gender bias,\" would the model introduce new artifacts or suffer reduced quality when generating \"landscape photos\" or \"animal pictures\"? This is a common challenge for all model editing methods, and the paper should provide a more detailed analysis and experimental results.\n3.Lack of In-depth Qualitative Evaluation and Repair Examples. Despite claiming that SURE is \"highly practical\" and offers significant speed improvements, the paper lacks detailed qualitative evaluation of the repaired images. It does not provide comparative examples of images before and after repair, nor does it discuss in detail the visual quality, semantic consistency, or the introduction of new artifacts in the repaired images. Relying solely on quantitative metrics (such as speed) is insufficient to fully demonstrate the effectiveness of the repair method. Providing actual examples of \"female CEO\" images generated after fixing the \"CEO gender bias,\" along with comparisons to the original problematic images, would greatly strengthen the argument's persuasiveness.\n4. The paper primarily discusses model bias, such as gender bias. However, the scope of \"failures\" can be very broad, including outdated information, factual inaccuracies, removal or addition of specific styles, etc. The paper does not adequately demonstrate SURE's ability to handle and repair diverse types of \"failures.\" For more complex or nuanced repair tasks, SURE's effectiveness remains unknown.\n5.The paper mentions that the Model Repair Institute (I) needs to maintain a \"repair database.\" However, there is no in-depth discussion regarding the practical scale of this database, its construction and maintenance costs, or how to ensure efficient retrieval and repair when the number of database entries becomes vast. As the types of model failures continuously increase, the scalability of the database and its associated computational and storage overhead could become a bottleneck for practical deployment. Furthermore, if multiple potential W_fix entries are applicable to the same query, or if different W_fix entries conflict, how does SURE handle these situations?\n6.The title \"Blindfolded Repair\" implies complete privacy for all parties involved. However, this \"blindfolded\" aspect may not hold true in certain real-world adversarial scenarios, such as when a malicious attacker actively deviates from the protocol to gain information, or in situations where user feedback itself contains highly sensitive Personally Identifiable Information (PII). The paper should more rigorously define the scope of privacy covered by \"blindfolded repair\" and the adversarial models it addresses.\n\nQUESTIONS:\n1.Questionable Generality of W_fix: How generalizable is the \"One fix fits all\" claim for W_fix across diverse repair tasks beyond simple bias correction (e.g., object removal, style alteration), and what experimental evidence supports its non-destructive application to such varied failures?\n2.Unevaluated Impact on Unrelated Generation Tasks: What is the impact of modifying all 32 cross-attention layers on the model's general generative capabilities for unrelated prompts, and do these global modifications introduce unintended artifacts or reduce quality for unaddressed tasks?\n3.Lack of In-depth Qualitative Evaluation and Repair Examples: Could the paper provide detailed qualitative evaluations, including before-and-after image comparisons, to demonstrate the visual quality, semantic consistency, and lack of new artifacts in repaired images, as quantitative metrics alone are insufficient?\n4.Insufficient Verification of Failure Type Diversity: How does SURE's effectiveness generalize to a broader range of failure types beyond bias correction, such as outdated information, factual inaccuracies, or stylistic changes, and what experimental evidence supports this versatility?\n5.Scalability and Maintenance Costs of the Repair Database: What are the practical implications and costs associated with scaling and maintaining the \"repair database,\" particularly concerning its construction, efficient retrieval, and conflict resolution when multiple W_fix entries apply or conflict?\n6.\"Blindfolded Repair\" Context and Limitations: Given the title \"Blindfolded Repair,\" could the paper more rigorously define the scope of privacy and the adversarial models covered, especially concerning malicious attackers or sensitive PII in user feedback?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SURE, which is a light-weight two-party protocol (2PC) that lets a model owner obtain a weight-patch matrix to fix a user-flagged error in diffusion model while keeping weights, user feedback, and the editing method hidden. In this protocol, the repair institute precomputes a fix matrix which the service provider can multiply it to all cross-attention layers locally, which enables a full model repair with small traffic and the protocol is proven secure in the semi-honest model.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n- The paper tackles a well-motivated problem that is relevant to real-world service providers of diffusion models, as it aims to fix errors in systems that are already deployed while complying to security constraints.\n- The paper provides tight security guarantees, with proofs in the hybrid model as well as their algorithm-hiding property of the protocol. \n- The method results in a substantially more efficient end-to-end repair efficiency compared to a baseline method with ~17s running time for a 3,000 sized database. \n\n**Weakness**\n- The paper does not discuss malicious-security guarantees or expected change of cost (compared to semi-honest setup) but this kind of setting might be required for a real-world use case.\n- The paper discusses only Stable Diffusion v1.4, which has a significant gap between more recent generation models that are typically deployed and might need repairs in a real-world setup. Runtime analysis for larger models such as SDXL or SD3 might be necessary to demonstrate generalizability.\n- The baseline might be too weak, which might make the comparison a bit unrealistic.\n\nQUESTIONS:\n- Have you run any quantitative or qualitative studies to show that a patched model actually corrects the user-flagged error without degrading other outputs?\n- How would the protocol/runtime/cost have to change to change to a maliciously secure setting?\n- Is there a possibility of a stronger choice of baseline other than a full linear-scan MPC?\n- Is it possible to apply several fixes to one prompt at a time? Does the cost increase in this case?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the confidential collaboration problem between a diffusion model service provider and a model repair institute. When a diffusion model service user submitted unexpected outputs to the service provider, it's important for the provider to identify and fix such misalignment. While it’s common for a service provider to seek collaboration from external third-party model repair institutes, it’s important to keep the confidentiality of both the provider's proprietary model and user feedback, as well as the repair institutes’ proprietary repair technique. This paper proposes such a secure repairing method, SURE, that encodes the prior Text-to-Image Model Editing (TIME) method into a secure two-party computation (2PC) protocol. The key insight is to identify a decomposition of the computation in TIME to factor out the heavy matrix-multiplication from the 2PC protocol. This paper provides security proof for the proposed method. This paper experimentally evaluated it against a vanilla TIME implementation in 2PC without optimizations, and the results shows that the proposed (optimized) method can significantly reduce the runtime and communication overhead in practice.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The problem is well-motivated. The problem of efficient and confidential collaboration between service providers and model repair institutes is interesting.\n\n- The idea of incorporating existing editing algorithms with the 2PC protocol to achieve confidential collaboration is novel to this community.\n\n- This work identified an effective optimization in encoding the TIME editing algorithm in a 2PC protocol, which significantly reduced the computation and communication overhead and made the protocol practical.\n\n## Weaknesses\n\n- The proposed framework lacks discussion of assumptions/requirements for a general editing algorithm to fit and gain the overhead reduction. Without such discussion, the proposed method is better to be viewed as an optimized 2PC extension to the TIME algorithm, but not as a general framework.\n\nQUESTIONS:\n- Regarding the efficiency, what are the assumptions on the editing algorithm to fit this framework to reduce the overhead?\n\n- Regarding the functionality, can this framework support more general repair requests, like align a prompt with a (batch of) images, and general repair methods, like a fine-tuning based method?\n\n- How can another editing method be plugged-in this framework? What are the requirements?\n\n- Can the protocol be extended so that the service provider does not lose access to prior repairs even if the model repair institutes stopped the service? For example, allowing the service provider to store the database and do the database matching offline?\n\n- Minor presentation suggestions: \n\n  - Regarding Table 1, it's recommended to place the caption on the top of a table.\n\n  - If possible, it might be more intuitive to align the unit of time and data for the baseline and SURE in Table 1 to highlight how significant the improvement is. For example, the time unit can be inlined like “167.36h” vs. “2.81s”. This is just a suggestion but not a demand.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}