RespoDiff: Dual-Module Bottleneck Transformation for Responsible & Faithful T2I Generation
Authors: Silpa Vadakkeeveetil Sreelatha
Sauradip Nag
Muhammad Awais
Serge Belongie
Anjan Dutta
Venue: NeurIPS 2025 poster ()
Keywords: Responsible T2I generation
Fairness
Safe generation
Debiasing
Diffusion models
Stable Diffusion
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: A novel framework for responsible text-to-image generation that incorporates a dual-module transformation on the intermediate bottleneck representations of diffusion models.

Abstract: The rapid advancement of diffusion models has enabled high-fidelity and semantically rich text-to-image generation; however, ensuring fairness and safety remains an open challenge. Existing methods typically improve fairness and safety at the expense of semantic fidelity and image quality. In this work, we propose RespoDiff, a novel framework for responsible text-to-image generation that incorporates a dual-module transformation on the intermediate bottleneck representations of diffusion models. Our approach introduces two distinct learnable modules: one focused on capturing and enforcing responsible concepts, such as fairness and safety, and the other dedicated to maintaining semantic alignment with neutral prompts. To facilitate the dual learning process, we introduce a novel score-matching objective that enables effective coordination between the modules. Our method outperforms state-of-the-art methods in responsible generation by ensuring semantic alignment while optimizing both objectives without compromising image fidelity. Our approach improves responsible and semantically coherent generation by \textasciitilde20\% across diverse, unseen prompts. Moreover, it integrates seamlessly into large-scale models like SDXL, enhancing fairness and safety. The project page is available at https://vssilpa.github.io/respodiff_project_page.

RespoDiff:
Dual-Module Bottleneck Transformation for Responsible & Faithful T2I Generation
Silpa Vadakkeeveetil Sreelatha' University of Surrey
Sauradip Nag Simon Fraser University
Muhammad Awais University of Surrey
Serge Belongie University of Copenhagen
Anjan Dutta
University of Surrey
Abstract
The rapid advancement of diffusion models has enabled high-fidelity and semanti- cally rich text-to-image generation; however, ensuring faimess and safety remains an open challenge. Existing methods typically improve fairness and safety at the expense of semantic fidelity and image quality. In this work, propose RispoDiff, a novel framework for responsible iext-to-image generation that incorporates a dual-module transformation the intermediate bottleneck representations of diffu- sion models. Our approach introduces two distinct learnable modules: one focused capturing and enforcing responsible concepts, such fairness and safety, and the other dedicated to maintaining semantic alignment with neutral prompts. To facilitate the dual learning process, introduce novel acore-matching objective that enables effective coordination between the modules. Our method outperforms state-of-the-an methods in responsible generation by ensuring semantic align- ment while optimizing both objectives without compromising image fidelity. Our approach improves responsible and semantically coherent generation by -20% across diverse, unseen prompts. Moreover, it integrates seamlessly into large-scale models like SDXL, enhancing fairmess and safety ,The project page is available https::/ 1pa. q thub. 1 re respodi ff. ff__rr_______________ prappeapppagee
Introduction
Models such as Stable Diffusion v1.4 (SDv1.4), SDXL, FLUX, and SD3 exemplify recent advance- ments in text-to-image (T2I) generation, revolutionizing content creation and visual communication by generating high-quality visuals from text prompts (Podell et al. 2024 Rombach al. 2022).
However, these models risk reinforcing stereotypes or producing harmful content, which can lead to societal consequences (Luceioni et al. 2023 Perera and Patel 2023 Rando et al. 2022 Schramowski
et al. 2023). Ensuring a responsible workflow is critical to mitigating these risks.
Previous methods for responsible text-to-image (T2I) generation include prompt modification (Chuang et al. 2023 Ni et al., 2024). model fine-tuning (Gandikota et al., 2023 [Shen et al.) 2024).
model editing (Gandikota al. 2024), , classifier guidance (Schramowski et al. (023). and latent
vector injection (Li et al, 2024). Despite advancements in responsible T2I generation, many existing approaches still compromise semantic fidelity and image quality, reducing their effectiveness in producing both responsible and faithful generation. To address these challenges, we propose RespoD- iff, novel framework for responsible T2I generation that introduces dual-module transformation on the bottleneck representations of diffusion models. Specifically, given a responsible category, such demographic attributes (e.g., gender) safety factors and its associated target concepts
Email address: macikkeeeeetiaeeeeeetaaaaaaaa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(e.g.."man", "woman" etc.), our approach learns independent transformations that steer the diffusion model toward target-aligned outputs while maintaining coherence with a neutral prompt (e.g.. "a person"). These learned transformations can then be applied during inference to promote fairness and safety T2I generation without compromising the underlying structure of the diffusion process. RispoDiff incorporates two learnable modules: @ Responsible Concept Alignment Module, which steers latent representations toward fair and safe outputs by learning transformations that align with responsible target concepts; and @ Semantic Alignment Module, which preserves consistency with neutral prompts, ensuring the generated images remain aligned with original prompts.
At the core of RespoDiff is score- matching objective that coordinates the two modules. For the demographic category "gender", given a neutral prompt (e.g., "a person") and a target concept (e.g.. "a woman"), "), we introduce an objective to guide the Responsible Concept Alignment Module. which learns to modify the diffusion trajectory of the neutral latent such that it closely approximates the trajectory associated with the target concept. A key aspect of our approach is leveraging the neutral denoised latent, obtained by passing the neutral prompt through the diffusion model. This serves as stable reference point, allowing predictions for the neutral and target concepts. This ensures that the transformation consistently steers generation toward the desired concept.
to extract explicit directional guidance by comparing UNet
Additionally. we introduce a score- matching objective to mitigate excessive influence from the learned transformation and prevent oversteering toward the target concept. This objective updates the Semantic Aligmment Module, ensuring that the dual-module transformation stays aligned with the original generative trajectory for the neutral prompt. The Semantic Alignment Module safeguards the structure and semantic details of the image with respect to the neutral prompt, thereby preserving visual fidelity of the original diffusion model. By jointly optimizing these objectives, our method ensures that the generated outputs align with the target concept while maintaining all other visual details consistent with the neutral prompt.
We empirically validate the effectiveness of our approach for responsible T2I generation, with a focus on fairness and safety. RespoDiff surpasses existing Mair-eenrration baselines and effec- tively generalizes to unseen prompts, including
feriession-speciffe scenarios, without requiring training or fine-tuning, Our approach further ensures semantic alignment
any profession-speciii with prompts while maintaining the visual quality of diffusion models. Additionally, it eliminates harmful or unsafe outputs without compromising image fidelity or alignment, demonstrating the practicality and robustness of our framework. Notably, our framework can seamlessly be integrated into scrge T2I models such as SDXL, enhancing faimess and safety in real-world deployments.
The key contributions of this work are as follows: We introduce RespoDiff, novel dual-module transformation for diffusion models, integrating a Responsible Concept Alignment module with a Semantic Alignment Module to ensure responsible generation while being faithful to the original diffusion process. @ We propose a simple score- matehing objective that enables effective coordination between the modules, ensuring seamless integration of responsible generation and prompt alignment. @ Our method achieves approximately 20% improvement in fairness and safety metrics while ensuring high semantic fidelity and image quality, demonstrating robustness across unseen prompts.
2 Related Work
T21 generation has transformed generative AI, enabling highly realistic image creation from text (Ho jet al. 2020; Ramesh et al. 2022, Rombach et al., (002)). but also raises ethical concerns, including the risk of generating harmful or inappropriate content (Cho et al. 2023 Luccioni et al., 2023).
Responsible Generation using Diffussin Models: In recent years, there has been growing emphasis on methods to reduce biased and inappropriate content generation in diffusion models, such as Stable Diffusion. Several approaches focus on modifying input prompts by removing harmful or
problematic terms (Ni et al. 2024 (Schramowski et al., 2023), while others employ prompt-tuning techniques (Kim et al.] (023) or learn projection embeddings on prompt representations (Chuang et al. (023) to filter out undesirable content. Some methods, such as those by Gandikota et al. (2023). Huang et al. (2024) and Zhang et al. (2024) attempt to erase unsafe concept representations from the diffusion models, though these techniques may negatively affect the model's original performance.
Similarly, (Shen et al. (2024) addresses biases by fine-tuning specific parts of the model weights, but such approaches require additional training for each prompt domain. In contrast, method

---

Neuual Preneet
Sermantic Alguaneat
Ctrue
Tree Naand Sortz
UNet
sem
Prompe
Lalis
Epsu
Noural Sconz
Ry
Uiff Emonder
RAM
bii UNE Dereeder
nomsen Prompe
ressp
Frezsn Modale
Leam Reggonsible Keneepes- Woman
CLar
Dillasion UNet
Backpopagttinn
Himzorpo
canesiop
Figure 1: Illustration of RespoDiff: RespoDiff performs reverse diffusion to timestep / using the prompt "a person", obtaining latent zt,acu as the neutral denoised latent for all forward processes. Forward diffusion with the RAM and "a person" predicts neutral score, with mean-squared error between neutral and target scores updating the RAM. To maintain faithfulness to the original diffusion process, forward diffusion with both RAM and SAM generates neutral score, with mean-squared eroor between neutral and original scores updating the SAM.
avoids prompt-specific fine-tuning and generalizes effectively across diverse and unseen prompts, including profession-specifis ones. Alternative strategies such as those by Friedrich et al. (2023). Parihar al. (2024), anhhramramw et al. (2023) use classifier-free guidance to steer the generation process away from undesirable content without requiring extra training. While some methods propose efficient closed-form solutions for embedding matrices to ensure responsible content generation (Chuang et al. 2023 Gandikota et al. (2024), they lack the adaptability and fine-grained control over
image generation offered by our approach.
Concept Discovery in Bottleneck Layer: Our method shares the goal of learning responsible concept representations in the latent space of diffusion models. [won et al. (202)) were among the pioneers to identify the bottleneck layer of U-Net (the h-space) as a semantic latent space, demonstrating that interventions within this space lead semantically meaningful changes in the generated images. Their approach uses off-the-shelf CLIP classifiers to learn disentangled representations in the h-space. Li di al. (2024) built this work by identifying interpretable directions in the latent space for target concepts. Their approach generates target concept images, adds noise, and then denoises them using neutral prompt and learnable vector optimized via noise reconstruction. However, it lacks explicit reference to the neutral denoised latent, relying on indirect supervision that may lead to less precise control over the transformation. Additionally, does not explicitly enforce faithfulness, risking unintended deviations. In contrast, RispoDiff directly models diffusion trajectory shifts using the neutral denoised latent for precise concept learning while integrating a Semantic Alignment Module to ensure coherence with the original prompt.
3 Preliminaries
In this section, we provide the necessary background regarding diffusion models and the scoring functions which form the foundation of our model design.
et al. 2015) are likelihood- based generative models inspired by nonequilibrium thermodynamics (Song and Ermon 2019). The model learns a denoising process that transforms random noise into samples from original data distribution, Piaaa The process involves gradually corrupting training data with Gaussian noise in forowardd process, where an initial sample zu ~ Pana is progressively noised into æ1, T2,..... TT through a Markovian process as follows:
Diffusion Models:
Diffusion models (Ho et al. 2020;
q( (z0) = II. ((20//=11))
g Naxxa / 8, 3.I).
3

---

where T is the total number of steps (typically 1000), and variance schedule 3r ensures 9r (æT) 2 (0 I). The reverse process learns reconstruct the original data by reversing this diffusion.
po (x p(=T)(
po (zt- (x)),
Po (2t-1 (2x.) = (a; ( (), oz1).
where ( t) is parameterized using a noise prediction network co (z, t). After training, genera- tion in diffusion models involves sampling from (o()) starting with noise sample zy p(ær) and recovering En Pdata using an SDE/ODE solver (e.g., DDIM (Song et al. (021a)). These models learn the transition probabilities ( (xt), defined follows:
Tt-1
co( + o We, wi W(O,I),
(I)
Vau
VI - at
where Oez, Ot and B, are predetermined noise variances, and w, time-dependent weighting function.
The noise prediction network (z( t) iteratively estimates the noise to generate Zy from 2y and approximates the score function (Ho et al. 2220)) Song et al. (021b), given by Va, logpt(z; pr (a:+) -eo (a,, 1)/at, where o, is the noise level time step and p: is the marginal distribution of the samples noised to time l. Following the score function direction guides samples back to the data distribution.
Diffusion Scoring Function:
T21 Diffusion Models:
T2I models generate images conditioned on text, using a UNet-based noise prediction network (0 y, t), where y is the text prompt. Rombach et al. (2022) use Latent Diffusion Models (LDMs), where diffusion operates latent space že instead image space Ty. Image generation starts by sampling latent noise ær, applying reverse diffusion obtain 2o, and decoding with VAE to obtain To. Classifier-fre free guidance (Ho and Salimans 2022) is used to enhance conditional generation by adjusting the score function follows.
(z(z y,t) Eo c 0,t) + (Z65 y, t) ez(zziy= Ø,))
where is the guidance scale and (z(2;; y = 1) denotes the unconditional score. The objective to train T2I diffusion models is given by:
Lan = Ez,,, [160(E)))))) - <16]
(3)
Method
We propose RespoDiff, a framework for responsible T2I generation via dual transformations on the intermediate representations of diffusion models. An overview is shown in Fig. I
4.1 Problem Formulation
We denote the UNet of pre-trained T2I diffusion model : 2 Z, where Z represents the latent space and y denotes the set of textual prompts. The UNet consists of an encoder e Zxy H. which maps latent z and textual prompt y an intermediate representation hy E H. and decoder H y + Z, which generates an updated latent z'. Formally, the model is expressed as:
f(z, v) = g(e(z,
(2 u). y) g(h,,, 8),
where represents the bottleneck space. Since the encoder and decoder always take the same text input in our setting, we represent g g(h 2) g(hr for simplicity. The final image is obtained by decoding using a VAE decoder.
To ensure responsible T2I image generation, we focus on a set of responsible concept categories, denoted as C = (Cyenter, Crace, Cufe). Each category comprises sensitive concepts, represented as S,: for example, Sgender {man, woman), Srace [black, asian, white), and Safe [violence, nudity) ). These categories and concepts are selected in alignment with prior works address fairness and safety concerns (Gandikota al. 2023 Li ct al. 2024).

---

We consider a neutral prompt, denoted as Yoeu, to facilitate responsible generation. For Cgender and Crace we use Yneu "a person", which provides general description of human subjects. For Csafe- we use Yneu = "a scene", enabling the model to generalize safety transformations across diverse scenarios. Additionally, target prompts corresponding to each concept & E Se are denoted as yor
Let neu E H denote the intermediate bottleneck representations of f corresponding to the neutral prompt Vnen. RespoDiff aims modify hnen such that the transformed model:
(ynee) g(T (Rnea) = ((T
resp, (hheu) + To To (hrea))
produces images aligned with the target concept s E Se. while ensuring semantic alignment and visual quality of original diffusion model f. The learnable transformation T/ is decomposed into two components where Tg ensures fairness and safety by aligning generated images with the intended target concept 8, and 7g preserves the semantic content and visual quality of the images.
By selecting general base descriptions, such as Vneu = "a person" or Yneu "a scene", the learned transformations are designed to generalize effectively across diverse human representations and scene contexts. For fairness, Vur corresponds to concept E S Sgender U Stace. For safety, following [i et al. (2024), we treat s E Saaee as negative concept, where the associated yūr serves as a negative prompt. This encourages the model to steer away from unsafe content by contrasting it with neutral prompt.
4.2 Responsible Concept Alignment Module
In this section, we introduce the Responsible Concept Alignment Module (RAM) Tg which modifies the latent representation hasu of a neutral prompt align it with a target concept. Given neutral prompt Yneu and a target concept prompt Viee the goal is to adjust the diffusion trajectory such that the neutral latent evolves reflect the target concept s.
To achieve this, we employ a score-maat matching objective that directly aligns the model's latent output with the target concept by manipulating the trajectory of the neutral latent. At a randomly selected timestep we begin by sampling the neutral denoised latent, 2t,ucu- from the diffusion model through reverse diffusion. This denoised latent becomes the starting point for its transformation towards the target concept. Subsequently, we compute the UNet predictions corresponding to the neutral prompt Yneu. denoted as Eyeu E//pp (Zt,acu- yuea) where fresp T(, (fmea) and
Aneu e c(Zz,mn Ineu). Using the same denoised latent Zt.neu. we compute the UNet predictions conditioned on the target prompt Virr represented as curr (2 Vir), where f ghuu)) and har e(zt,neu: Viar which serves as the ground truth predictions. The objective is formulated as:
Cresp = Ezz,, [Ile (Ft, Vneu) - E (Zt,,neu Menr 1
important to note that during this stage, the loss computation considers only fresps and not f. as the sole objective is to steer the RAM towards aligning with the target concept. Following [Song] ct al. (b021b), the objective in Eq. 44) can be viewed as minimizing the gradient disparity between the neutral and target diffusion trajectories, effectively guiding the latent space toward the target concept at each timestep. As a result, optimizing this score-matching loss enables the RAM to transform the latent representation such that the generated images are effectively aligned with the desired target concept. Unlike prior methods that rely on implicit reconstructions using pre-existing target images, our loss function explicitly supervises the transformation in the model's latent space by precisely aligning the diffusion trajectories of neutral and target concepts. By leveraging the neutral denoised latent as stable reference point, our approach ensures that the modification is structurally guided rather than inferred indirectly from external data.
4.3 Semantic Alignment Module
In this section, we introduce the Semantic Alignment Module (SAM) T.em,, which ensures that the transformations applied to the latent representations preserve semantic fidelity with respect to the original pre-trained diffusion model f. While RAM focuses on aligning with a target concept 8, the Semantic Alignment Module maintains consistency with the original diffusion trajectory.
5

---

To achieve this, we propose a score-matching objective that regularizes the transformation by preserving alignment with the original generative trajectory of neutral prompt. Specifically, we utilize the denoised latent representation Zr.ncu obtained from the reverse diffusion process f at a randomly selected timestep t. Using the default pre-trained diffusion model f, we compute the UNet prediction corresponding to the neutral prompt V/ecu denoted as frue = e ( Meeu) where g(hae)) and hneu e(zi,neu: Mneu). This prediction serves as reference for the original diffusion process. We then introduce a score- mattcing objective formulated as:
[|eg(Er,auu : Ey (Zt, Vreu
Syem
This loss penalizes any significant divergence between the steering of the Responsible Concept Alignment Module and the original diffusion model's behavior. By optimizing Leem. the Semantic Aligmment Module ensures that the generative process adheres closely to the original pre-trained model's trajectory, thereby preserving semantic fidelity and preventing the introduction of artifacts or unintended deviations.
4.4 Training and Inference
The training process alternates between optimizing the transformations Te and Tg to achieve responsible image generation while maintaining semantic fidelity. We do not backpropagate through the reverse diffusion when obtaining 2r,neu keeping the modules frozen primarily to avoid the computational overhead associated with it. In the first step, we update Tge by using the model frea g(TT images towards responsible concepts while maintaining consistency with the original diffusion process. In the second step, we switch to the full transformation f g(T,epp. (hnou) + To (hrec)) and optimize To alone using the semantic loss Leem weighted by x, where is a hyperparameter. As the iterative optimization progresses, the neutral denoised latent 2t,ecu, obtained via reverse diffusion through with learnable transformations, progressively aligns with both the target concept and the original diffusion process, leading convergence.
(hneu)) and minimizing the responsible loss Leep. This update steers the generated
During inference, for responsible generation with respect a category E C, we learn transformations Tj corresponding to each sensitive concept s Sc. To ensure fairness, we randomly select a concept Se and apply the corresponding transformation T during generation. This ensures that the output distribution remains uniformly balanced with respect to C. For safe generation, we aggregate the transformations across all concepts in Swfe, which includes violence and nudity since they broadly cover other safety concerns, such as shocking, harassment, and other harmful visuals. These transformations specifically correspond "anti violenvioeeeee" and "anti-sexual", learned through negative prompting with respective target prompts. The aggregation is performed by computing the summation EneSEe Th (h). This aggregated transformation is then used to generate images that mitigate inappropriate content while preserving semantic fidelity during inference.
5 Experiments
We evaluate the effectiveness of the learned responsible concepts in promoting fair and safe image generation. All experiments are conducted using Stable Diffusion v1.4 (Rombach et al. 2022) and Stable Diffusion XL (Podell et al., 2024) to assess the efficacy of our approach. Note that the boldfaced values indicate the best results, while the underlined values represent the second-best results across all evaluations.
5.1 Fair Generation
Evaluation Setting: We evaluate our method on the Winobias benchmark (Zhao et al. 2018).
following the approaches (Gandikota et al., 2024; Li et al. 2024 Orgad et al.) 2023). which includes 36 professions with known gender biases. We learn transtormations as constant functions linearly added to bottleneck activations, optimizing over 5000 iterations with batch size 1, as in and Section 4.2 Unlike Gandikota et al. (2024) and Shen et al. (2024), our approach does not rely on profession-speeifie data during training. Instead, we utilize the prompt "a person" to learn generalized directions that are applicable across professions. For fair comparison, adopt the experimental

---

2024) to evaluate fairness. Five prompts per profession are used, including templates like "A photo of (profession)". We also extend our evaluation the Gender+ and Race+ Winobias datasets (Li et al. 2024), which introduce terms like "successful" to trigger stereotypical biases (Gandikota al. 2024). Additional dataset details are in Section 8.4. of Appendix.
setup from Li et al.
Metrics: We perform quantitative and qualitative analysis to evaluate the performance of our proposed approach. We employ the modified average deviation ratio (DevRat), as defined in [i et al. (2024) to quantify the fairness of the generated images. We also measure the alignment of the generated images with the Winobias prompts (Wino-Align). Additionally, we assess image fidelity using the FID score (Heusel et al.) 2017) on the COCO-30K validation set (FID (30K)), while image-tex text alignment is measured with the CLIP score (Radford al. [20TT] using COCO-30K prompts under fair transformations (CLIP (30K)). Further details are provided in Section 8.4.6 of Appendix.
Firefighter
CEO
Baker
Cleaner
Nurse
Librarian
Designer
SD
Ours
CEO
Cashier
Mechanic
Const. worker
Doctor
Physician
Farmer
Ours
Figure 2: Comparison of RespoDiff and SD in generating profession images by gender (top: women in first 4 columns, men in the rest) and race (bottom: Black in first 2, Asian in next 3, White in last 2). RispoDiff better reflects target attributes while maintaining fidelity to SD outputs.
Table 1: Comparison of gender faimess, align-
Table 2: Comparison of race fairesss, alignment and quality across with SD v1.4.
ment and quality across with SD vl.4.
Approach
DevRat
Wineillia
FI)
CLIP
Approach
DevRat
WineAlign
(.)
CLIP
(CVPR. 2023g
0.68
27.51
14,09
31.33
(CVPR, 2022)
27.51
14:00
SDise (CVPR, 2024)
0.12
26.61
23.59
29.94
SDisc (CVPR, 2004)
0.23
26,80
17.45
30.27
FDF (ELLR, 2024)
0.50
23.90
15.22
30.65
FDF (CCLR, 2024)
0.32
23.15
14:94
30.59
BAet (CVPR, 2024)
0.57
27.67
17.07
30.54
BAet (CVPR, 2024)
0.45
27.63
17.20
30.47
RespoDiff
0.14
27.30
18.91
30.67
RespoDift
0.16
27.53
12.82
31.02
Results: We compare our approach against baselines including Stable Diffusion (SD) (Rombach
al. 2022). FDF (Shen al.] (2024), SDisc (Li al. 2024) and BAct (Parihar et al. 2024) with additional details in Section 8.4.2 Tables [ and present comparison of our approach to various focusing on various metrics across both gender and race biases. We further analyze extended biases within these categories, with additional comparisons provided in Section 8.4.7 RespoDiff consistently achieves the lowest average deviation ratio both gender and race biases. even in challenging settings, highlighting its superior performance in mitigating biases across different professions. As observed Section 8.4.8 of the Appendix, our method effectively eliminates gender and racial biases in a range of professions compared to Stable Diffusion. Although FDF performs better in certain professions like Secretary, likely due to training on profession-spectes images, our approach improves fairness across all professions average without being explicitly trained on
baseline methods, focusing
7

---

cerfession-speciifee prompts. This highlights our model's strong generalization ability across different professions, Additionally, RespoDiff remains robust to bias-inducing prompts with phrases such as 'successful", as detailed in Section 8.4.7
We also provide additional comparisons in Section [8.4.3
Our approach achieves strong text-to-image alignment (WinoAlign) while maintaining the most balanced gender representation across fair concepts. Effective debiasing should preserve image fidelity and text alignment, particularly for non-stereotypical prompts which evaluated using compute FID (30K) and CLIP (30K). As shown in Tables I and 2
RespoDiff maintains image quality comparable Stable Diffusion across gender and race debiased models while ensuring strong text-to-image aligmment. Unlike SDisc, which operates in the bottleneck space but struggles with image quality, our approach effectively balances fairness without compromising generation quality.
Qualitative analyses in Fig. 2 further support our quantitative findings. RespoDiff effectively mod- ifies Stable Diffusion outputs align with sensitive subconcepts like "woman". "black race" etc. using learned transformations while preserving profession-specific attributes. Additional results are available in Section 8.9 We also present additional qualitative comparisons of our approach with hard prompting and SDisc in Section 8.4.4 Section 8.4.5 respectively.
5.2 Safe Generation
The training process conducted for 1500 iterations with a batch size of in the safe generation experiments. During evaluation. generate images using prompts from the I2P benchmark (Schramowski et al. 2023), which consists of 4703 inappropriate prompts categorized into seven classes, including hate, shocking content, violence, and others.
Evaluation Setting:
Metries: To assess inappropriateness, we utilize a combination of predictions from the Q16 classifier and the NudeNet classifier on the generated images, in line with the approaches presented in Gandikota jet al. (2023); Schramowski et al. (2023); ; Li et al. (2024). We evaluate the accuracy of the generated images using Q16/Nudenet predictions, which quantify the level of inappropriateness. We also compute the FID and CLIP scores to assess image fidelity and image-text alignment using the COCO- 30K prompts. We compare our approach against baselines such as SD, SDisc, ESD (Gandikota et al. 2023) and SLD (Schramowski et al. 2023). Further details are provided in Section 8.5.2
CS
Ours
Figure 3: Qualitative comparison of safe generation. RespoDiff removes nudity and violence present in SD outputs, producing safer and more appropriate images.
Results: Table 3 compares average Q16/NudeNet accuracies across all seven I2P benchmark classes, where our approach Despite training on a limited safety set State = {violence, nudity). RespoDiff generalizes well to other safety-critical categories, as detailed in Section 8.5.3 Qualitative results are shown in Fig. 3 We further assess FID and CLIP scores, demonstrating that our approach maintains im- age quality comparable to Stable Diffusion on COCO-30K while achieving superior image-text alignment. Notably, our method not only en- hances safety beyond SLD,, which designed specifically for safety, but also outperforms it in image-text alignment. While ESD and SDisc achieve higher image quality, approach effectively
surpasses
baselines by 20% margin.
Table 3: Comparison of safety and image quality metrics across approaches with SD v1.4.
Appronch
12P (L) (30K) (L) (30K)
SD Rombach 2022
0.27
14.09
31.33
SDisc (Lre 2014
0.27
15.96
31.03
Scaring
18.76
ESD Ganorsola et 2023
0.32
13.68
30.43
RespoDift
0.16
17.89
31.10
8

---

Table 4: Comparison of gender faimess, align- ment and quality metrics with SDXL.
Approach
DevRat (J)
WinoAlign (1)
FTD (i)
CL.PP (1)
28.50
13.68
32.19
RispoDiff
0,26
28.02
14.63
32.03
Care
Table 5: Comparison of race fairness, alignment and quality metrics with SIXI..
Approuch
DevRat (1)
WinoAlign (1)
CLIP (1)
Figure 4: Comparison RespoDiff and SDXL
SD
0.57
28.53
13,68
32.19
for the Doctor profession, steered towards: Woman, Black, Asian and White (left to right).
RespoDiir
32.04
filters inappropriate content without significant degradation in visual fidelity. Additional results provided in Section 8.9
5.3 Responsible Generation using SDXL
In this section, we explore the applicability of RespoDiff large-scale generative models, specifically SDXL, for responsible T2I generation. We train our method using SDXL for the fair generation task and evaluate its effectiveness by comparing the results against the baseline SDXL model. The quantitative results are reported in Table Table We also train RespoDiff using SDXL for safe generation and the results are provided Section 8.5.4
As shown in Table 4 and Table 5] RespoDiff significantly mitigates biases related to gender and race that are present in SDXL, achieving substantial improvements. Importantly, our method preserves both the visual quality and image-text aligmment of the original SDXL model. We also provide qualitative results demonstrating the transformation of concepts such as woman, Black race, Asian race, and White race in Fig, 4 and Section 8.9 of the Appendix. Notably, RespoDiff accurately steer generated images toward target concepts while maintaining consistency with the original content.
5.4 Ablations
We perform ablation studies analyze RespoDiff, examining the impact of individual modules and the benefit of employing separate modules for responsible concept alignment and semantic alignment. We also assess the influence of different architectures for these modules in Section 8.6
Table 6: Effect of different modules used in our approach.
Approach
WinoAlign (t)
FID (30K) (L)
CLIP (30K) (1)
RAM
0.12
26.12
15.63
29.93
RAM SAM (RespoDiff)
0.14
27.30
14.91
30.67
Effect of Individual Modules: We analyze the impact of the RAM and SAM modules on fairness and image quality. The results summarized in Table 6 RAM alone yields lower deviation ratio, indicating reduced bias, but slightly compromises alignment (lower CLIP and WinoAlign scores). Adding SAM increases the deviation ratio but improves both image quality and alignment, highlighting its role in balancing fairness with generation fidelity.
Table 7: Ablation on the effect of using single transformation for both responsible and semantic alignment.
Approach
DevRat (L)
WinoAlign (1)
FID (30K) (L)
CLIP (30K) (1)
Shared module (single T
0.16
26.12
15.63
29.93
Separate modules (RespoDift)
0.14
27.30
14.91
30.67

---

Effect of Shared Transformation: In this section, we assess the impact of replacing the default RespoDiff setup, which uses two separate transformations for responsible and semantic alignment, with a single shared transformation T trained using the combined loss Lessp + ALrem- As shown Table 7 the shared variant results in a lower deviation ratio with better alignment and fidelity metrics. We attribute this improvement to dedicated transformations, which streamline concept and semantic alignment learning, allowing each to focus on its task independently for more effective model optimization.
Table 8: Sensitivity of RispoDiff to the hyperparameter A.
DevRat (4)
WinoAlign (f)
FID (30K) (L)
CLIP (30K) (t)
0
0.12
26.12
15.63
29.93
0.5 (Default)
0. 14
27.30
14.91
30.67
4
0.29
27.53
14.17
31.24
Sensitivity to A: In this section, we analyze the sensitivity of RespoDiff to the À hyperparameter which is used control the weight of SAM. As observed in Table increasing À keeps the trajectory closer to the base model, improving fidelity but weakening target-concept steering, while decreasing has the opposite effect. We therefore use 0.5 as a knee point that balances fairness and fidelity.
Conclusion
Our proposed framework, RespoDiff, presents a significant advancement in responsible text-to- image generation by effectively ensuring faimess and safety of content generation. By leveraging dual-module bottleneck transformation and a novel score-matching objective, our approach ensures responsible generation without degrading image quality. Extensive evaluations demonstrate its superior performance over existing methods, achieving a substantial improvement in generating fair and safe images across diverse prompts. Furthermore, integration of RespoDiff into large-scale models like SDXL highlights its practical applicability and scalability. This work represents crucial step toward the development of more reliable T2I generative models, paving the way for safer Al-driven creativity.
7 Limitations and Future Works
Our work tackles the mitigation stage of responsible generation by introducing modular trans- formation framework that balances fairness with fidelity. While effective, this approach assumes that fairness and safety concepts are specified in advance, a design choice shared with prior works. This ensures clear evaluation and fair comparison, though it confines mitigation of predetined cat- egories alone. A natural extension is address unseen or emergent concepts not included in the original training set. The modularity of RespoDiff makes this practical: each concept corresponds to a lightweight transformation, so new modules can be added without altering the backbone. Furthermore, intersectional biases can be mitigated without additional training by composing existing modules, which partially addresses unseen identities. Nonetheless, automatic discovery of emergent concepts remains an important and orthogonal challenge. One promising direction is to leverage world models or LLL--bssed procedures (D'Incà et al. (2024) to propose new categories, after which RespoDiff can readily learn the corresponding transformations.
Our framework also assumes access a well-defined neutral prompt. For fairness, we adopt "a person", which generalizes well across diverse human contexts, and for safety we use "a scene", which captures a broad range of environments. To test robustness this assumption, additional experiments (Section 8.7p confirm robustness under alternatives such as "a group of people". For real-world deployment, would be valuable to automate neutral prompt identification. A practical strategy is to generate small pool of candidate prompts using LLMs, train RespoDiff modules on this pool, and at inference select the most appropriate one by measuring similarity between the user prompt and the candidates. For example, a prompt "a group of friends on picnic" aligns most closely with the neutral prompt "a group of people". Preliminary results provided Section (.88 validate the fcasibility of this similarity- based strategy, and expanding it remains an interesting direction.
10

---

Acknowledgements
Serge Belongie and Silpa Vadakkeeveetil Sreelatha were supported in part by the Pioneer Centre for AI, DNRF grant number Pl. Sauradip Nag acknowledges support from the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant. Silpa Vadakkeeveetil Sreelatha also thanks the ELLIS PhD Program for support and acknowledges travel support from the ELIAS Mobility Fund and the Turing Mobility Scheme (2024/25) from the UK.
References
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In ICCV, 2023.
Ching-Yas Yao Chuang. Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debiasing vision- language models via biased prompts. arXiv, 2023.
Moreno D'Incà, Elia Peruzzo, Massimiliano Mancini, Dejia Xu, Vidit Goel, Xingqian Xu, Zhangyang Wang. Humphrey Shi, and Nicu Sebe. Openbias: Open-set bias detection in lext-to-image generative models. In CVPR, 2024.
Felix Friedrich, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Patrick Schramowski. Sasha Luccioni, and Kristian Kersting. Fair diffusion: Instructing lext-to-image generation models faireess. arXiv, 2023.
Rohit Gandikota, Joanna Materzyúskk,, Jaden Fiotto-Kaufman and David Bau. Erasing concepts from diffusion models. In ICCV, 2023.
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyúika,, and David Bau. Unified concept editing in diffusion models. In WACV, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurlPS, 2017.
Jonathan Ho and Tim Salimans. Classificr-free diffusion guidance. In DGM-DA, NeurIPS, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Chi-Pin Huang, Kai-Po Chang, Chung-Ting Tsai, Yung-Hsuann Lai, Fu-En Yang, and Yu-Chiang Frank Wang. Receler: Reliable concept erasing of lext-to-image diffusion models via lightweight erasers. In ECCV, 2024.
Eunji Kim, Siwon Kim, Chachun Shin, and Sungroh Yoon. De-stereotyping text-to-image models through prompt tuning. In JCMLW, 2023.
Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have semantic latent space. In ICLR, 2023.
Hang Li, Chengzhi Shen, Philip Torr, Volker Tresp, and Jindong Gu. Belf-discovering interpretable diffusion latent directions for responsible text-to-image generation. In CVPR, 2024.
Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. Stable bias: Evaluating societal representations in diffusion models. In NeurIPS (Dutasets and Benchmarks), 2023.
Minheng Ni, Chenfei Wu, Xiaodong Wang, Shengming Yin, Lijuan Wang, Zicheng Liu, and Nan Duan. Ores: Open-vocabulary responsible visual synthesis. AAAI, 2024.
Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov, Editing implicit assumptions in text-to-image to--too-mmaee diffusion models. In ICCV, 2023.
Rishubh Parihar, Abhijnya Bhat, Abhipsa Basu, Saswat Mallick, Jogendra Nath Kundu, and R. Venkatesh Babu. Balancing act: Distribution-gun guided debiasing diffusion models. In CVPR, 2024.
Malsha V. Perera and Vishal M. Patel. Analyzing bias in diffusion- based face generation models. In IJCB, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL. Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.

---

Aditya Ramesh. Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv, 2022.
Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramer. Red-teaming the stable diffusion safety filter. ML. Safery, NeurIPS, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting, Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In CVPR, 2023.
Xudong Shen, Chao Du, Tianyu Pang, Min Lin, Yongkang Wong, and Mohan Kankanhalli. Finetuning text-to-image diffusion models for fairness. In ICLR, 2024.
Jascha Narain Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, ICML, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients the data distribution. NeurlPS, 2019.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Gong Zhang, Kai Wang. Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learning forget text-to-image diffusion models. CVPRW, 2024.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-We Chang. Gender bias coreference resolution: Evaluation and debiasing methods. In NAACL, 2018.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Refer Section I
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Refer Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
12

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specing well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper do not have thcoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Refer to Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
13

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The link to the project page is provided in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nigg) cc/
public/ http://udddee//eeeeeeeee
ssionpolccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NewrIPS code and data submission guidelines (https)
/ ctnis.cccppbbll htoplllc/uuieeeeeeooeeeeeee des CodeS Smb ss i iooooooll i ccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
14

---

Answer: [Yes]
Justification: Refer to Section S
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The results averaged across 2 random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma eгrоr bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Refer to Section 8.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
15

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:httww....a.......oooo////tt
Answer: [Yes]
Justification: Sensitive content has been obscured with black squares and discussion on limitations and broader impact is provided in Section 8.2
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Refer to Section 8.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
16

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper uses existing public datasets and models and are properly credited wherever required throughout the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, |papersw wwwersswwthoee com/ dat aset s has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
17

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method do not involve any LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp///eeiipppp/eee ceteuurips.cc/Cofffeeee cc/Conffreenee/2225
LLM) for what should should not described.
18

---

8 Appendix
In the primary text of our submission, we introduce RespoDiff, novel framework that learns responsible concept representations in the bottleneck feature activations of diffusion models. To ensure our manuscript's integrity, we provide extensive appendix designed to complement the main text. This includes series of additional experiments, comprehensive implementation protocols, qualitative analyses, and deeper analyses of our findings. The Appendix is presented to bridge the content gap necessitated by the page constraints of the main manuscript, providing a detailed exposition of our methodology and its broader impact on the domain.
8.1 Limitations
Our method is built around binary and limited gender and racial categories, such as "man" and "woman" limited set of racial groups. which constrains its ability accurately represent more nuanced, non-binary, or intersectional identities. For instance, the model may struggle to generate outputs that represent individuals who identify as gender non-conforming racially mixed. Addi- tionally, the approach relies on a predefined set of concepts. If concept is not explicitly included during training like a specific gender identity such as "transgender", our approach fails to generate outputs related to that concept. This reliance on fixed categories may unintentionally reinforce narrow and incomplete representations of gender and race, potentially marginalizing underrepresented or emerging identities, such as those of trans and non-binary people, or individuals from newer racial categorizations. Furthermore, while our evaluation focuses primarily on professions such as doctors, engineers, or firefighters where biases are known to exist, our approach may still produce contextually inappropriate outputs in other domains when applied broadly. This can lead to unrealistic or inaceu- rate depictions. Additionally, despite our dual-module design, which attempts to balance fairness and semantic fidelity, residual biases may still emerge, particularly cases where definitions of fairness and neutrality are not clear-cut.
8.2 Broader Impact
Our work addresses the pressing need for responsible generative models by introducing a method, RespoDiff that steers diffusion-based image generation toward more fair and safe outputs, while preserving the semantic integrity of the input prompt. By enabling more equitable representations in image generation, particularly in professional RespoDiff can help counteract stereotypes in media, datasets, and downstream AI applications. Our work has the potential to drive positive societal change by contributing to the development of fairer and more inclusive generative models.
societal roles where bias is often amplified,
However, there are certain ethical considerations tied to the limitations of our method. The reliance on binary gender categories and fixed racial groups may reinforce normative societal frameworks, inadvertently excluding or misrepresenting non-binary, multiracial, or intersectional identities. Simi- larly, while effective at reducing inappropriate content, it may struggle with nuanced harmful imagery. Furthermore, fairness is complex and context-dependdnt concept. While our method can promote diversity and faimness within well-defined contexts (e.g.. professional roles), its application broader, less studied domains may lead inaccurate or inappropriate outputs. In summary, while RespoDiff presents a technical advancement in promoting responsible image generation, it must be deployed with awareness of its limitations and the potential for unintended consequences.
8.3 Pseudocode
We provide a pseudocode for the training and inference of RespoDiff in Algorithm aan Algorithm 2 respectively. We also provide an inference diagram in Fig. 5
8.4 Fair Generation
In this section, we discuss the datasets and some additional experimental details which include the qualitative analysis. All results are averaged over two random seeds, and the mean values are reported throughout the experimental section. We conduct all our training and inference experiments for Stable
19

---

Algorithm Training of RespoDiff
Input: (1) Pre-trained T21 diffusion UNet f : X with text encoder 2 x y - H. and image decoder : H. x y + 2; (2) Concept category C; (3) Sensitive concept $ (e.g. "a woman"); (4) Neutral prompt Uneu (e.g., "a person"); (5) Target prompt yür ("a woman"); (6) Hyperparameters: learning rate and weight for semantic alignment loss A.
Output: Updated diffusion model y X' ensuring responsible and faithful T2I generation.
1: Initialize T and T
2: while training is not converged do
3: Sample / ~ U(0, 50)
4: Sample initial latent 21 ~ (O(11)
5:
Reverse diffusion from 2T to using (1nee) to obtain neutral denoised latent nenu
6:
Compute Encu Efo (Zr, Bucu) where fresp ((Tg ((hacu)) ) and hocu el pocu Uncu)
7:
Compute Cur = (2t,neu Virr), where f = g (howr) and h = c( e( Varr);
8:
Compute responsible loss in Eq. (4)
9:
Update T, using gradient descent: To f- To -n Sum Cresp
Compute Enne = (ft Moen where f g(T (henc)) + TT s.sm (Fanea));
10:
11:
Compute semantic loss in Eq. (5)
12:
Update To using gradient descent: To t To nAV f
13:
end while
Algorithm 2 Inference of RespoDiff
Input: (1) Neutral prompt Mneni (2) Target concept s (2) Updated diffusion model f y + Output: Image zo that aligns to target concept without much deviation from y/eu.
1: Sample 2e (O, 1)
2: for 1 do
It-1 ai (t Bee (2.xx.).))
Ciey Br are predefined scheduling parameters
4: end for
5: Return 350
Diffusion v1.4 on single NVIDIA RTX 3090 with 24 GB of VRAM while we use single A100 with 80GB memory for experiments with SDXL.
8.4.1 Dataset
We evaluate our fair concept transformations using the Winobias benchmark dataset, which comprises 36 professions. The list of professions includes: Attendant, Cashier, Teacher, Nurse, Assistant, Secretary, Cleaner, Receptionist, Clerk, Counselor, Designer, Hairdresser, Writer, Housekeeper, Baker, Librarian, Tailor, Driver, Supervisor, Janitor, Cook, Laborer Construction Worker, Developer, Carpenter, Manager, Lawyer, Farmer, Salesperson, Physician, Guard, Analyst, Mechanic, Sheriff. CEO, Doctor.
The five prompt templates are as follows: "image of <profession>", "picture of <professi on>". "photo of <profession "headshot of cprofession "portrait of sprofessi on>". For each prompt, we generate 5 images, resulting in 30 images per profession. In total, we evaluate 5400 images.
We also evaluate our approach on the extended Winobias dataset where the prompt templates are follows: "Image of a successful <profession>" "Picture of successful <profession>", "Photo of successful <professions "Headshot of successful <profession>", "Portrait of successful <profession>".
8.4.2 Baselines
We compare the performance of our proposed approach against several baselines: (1) Stable Diffusion (SD) (Rombach et al. 2022), samples are generated using the original Stable Diffusion model; (2) FDF (Shen et al. 2024), fine-tunes the text encoder of diffusion models using distributional
20

---

"A doctor"
Gender
Doctor
SD UNet
"A doctor"
Probabitty
Fair Gender Doctor Generation
Female
Male
RespoDiff modules
RespoDiff modules
Figure 5: Inference with RespoDiff. Stable Diffusion (top) produces biased generations for prompt "a doctor", predominantly depicting male doctors. RespoDiff (bottom) applies concept- specific RAM and SAM modules at inference, sampling across gender with equal probability, and yields balanced outputs while preserving semantic fidelity to the prompt.
alignment loss; (3) SDise (Li et al. 2024). learns concept vectors in the h-space using generated images; (4) BAct (Parihar et al. 2024) ensures uniform distribution of sensitive categories in the h-space where the psuedo labels from pretrained classifiers
h-vectors are utilized to guide the generation. We utilize the pretrained models provided by the authors for FD, FDF and BAct, whereas the results for SD, SDise are directly taken from the tables provided by the authors of SDisc. We chose these baselines because they are closest to our setting, where they either fine-tune the model or utilize the h-space for fair generation. These methods share commonalities with our approach, allowing for meaningful comparisons across various techniques designed to enhance fairness and model performance.
FDF (Shen et al. 2024) targets the mitigation four racial biases White, Black, Asian, and Indian whereas, in case, along with other baselines, focus reducing racial biases across three classes White, Black, Asian, following Li et al. (2024). Nevertheless, we employ the pretrained models released by the authors to evaluate their approach Winobias prompts for both Race and Race+ extended categories. Importantly, we ensure that their approach is evaluated using four CLIP attributes corresponding the racial classes they considered. Given that the deviation ratio metrie is designed to quantify fairness in generated images, we believe this constitutes a fair comparison.
8.4.3 Additional Baselines
We also compare our approach with methods that, although different from our setting, aim for the same goal of fair generation. For instance, (1) FD (Friedrich et al. 2023) directs the generation towards the target concepts while distancing it from other concepts by extending classifier guidance in diffusion models. The results are presented alongside those the main paper for simplicity in Table / and Table 10 Our findings show that our approach achieves better fairness with respect to gender and race, while maintaining alignment and quality, compared to FD.
8.4.4 Comparison with Prompt Modification
This section aims to evaluate the limitations of prompt-based fairness interventions that rely on adding explicit gendered descriptors, and to highlight the advantages of RespoDiff, which achieves fairness through latent-space interventions rather than textual prompt modifications, We perform qualitative comparison using the prompt "a photo of an engineer" (Fig. 6). The first row shows outputs from
21

---

Table 9: Comparison of gender faimess, align-
Table 10: Comparison of race fairness, alignment
ment and quality acToss with SD vl.4.
and quality across with SD v1.4.
Approach
DevRat
WinoAlign
(.)
(1)
Approach
WinnAllin
(.)
CLIP
(CVPR, 2022)
0.68
14.09
31.33
CVPR, 2022)
0.56
27.51
14,09
31.33
SiMse (CNPR, 2024)
0.12
25.61
23.59
29.94
Siise (CNPR 2024)
0.25
26.80
17.47
30.27
FDF (CLR, 2024)
0.40
23.90
15.22
30.63
FDF (CCL 2024)
0.32
23. 1.15
14.94
30.59
BAct (CVPR, 2024)
0.57
27.67
17.07
30.54
BAct (CVPR. 2024)
0:45
27.63
17.20
30.47
FD (Sc1. Adv. 2025)
0.31
22.61
15.56
31.80
FD (ScL. Adv. 2025)
0.50
27.59
15.54
30.82
RespoDilf
0.14
27.30
14,91
30,67
RespoTiit
0.16
27.53
12.82
31,02
Stable Diffusion using the baseline prompt. The second row illustrates the results of hard prompt modification, where we explicitly add gendered language (e.g., "a photo of a female engineer") to steer generation toward women. The third row presents results from our method, RespoDiff, which applies a latent transformation representing the concept "a woman" directly in the model's bottleneck representation, without changing the original prompt.
Most generated individuals by Stable Diffu- sion are male, revealing the gender bias embed- ded in the pretrained diffusion model. Hard prompt modification (second row) increases fe- male representation but introduces new biases: the women are frequently depicted in stereotyp- ical settings, such as working indoors, using laptops, or being placed in less technical envi- ronments. These outcomes reflect underlying training data biases that are activated by explicit gendered prompts. In contrast, RespoDiff (third row) promotes gender diversity while preserving the semantic intent of original image in realis- tic and technically appropriate settings. This comparison illustrates that while hard prompt modification can adjust demographic represen- tation, they often amplify latent stereotypes and distort scene semanties. RespoDiff, by oper- ating in latent space, provides a more robust alternative: maintains the integrity of the orig- inal prompt while enabling controlled and fair
Stable Diffusion
Hard prompts
RespoDiff
Figure 6: Comparison of RespoDiff and hard prompt modification for the prompt photo of an engineer."
concept steering.
8.4.5 Qualitative Comparison with SDise
In this section, provide qualitative comparison with SDise (Li et al.) 2024) to showcase the effectiveness of our method in generating fair and semantically accurate images, despite both methods operating within the bottleneck space of diffusion models. To evaluate this, apply both methods to the prompts "a photo of a Analyst" and "a photo of a Cashier," steering generation toward the concepts "a woman" and "a man". respectively. As shown in Fig. 7 both methods steer towards the target concept, but SDise often introduces artifacts or scene changes that compromise the intended profession, such unrealistic outfits or unclear office settings. Additionally, images of women are often overfitted to specific stereotypes (e.g.. sad or elderly faces), and depictions of men in certain professions (e.g., a cashier) can be unrealistic. We believe the shortcomings SDisc stem from its approach of directly reconstructing the noise, which biases the generation towards specific visual stereotypes and hence struggles to generalize to broader, contextually accurate representations of professions. This results in the overfitting of gender-specift traits or exaggerated features that misalign with the intended semantic context. In contrast, RespoDiff maintains the profession-specifs context while steering toward the intended concept. This highlights the effectiveness of the novel scoe-matching technique using RAM and SAM in our approach in achieving fair representation without compromising semantic fidelity.
22

---

SD
SDise
RespoDiff
SD
SDisc
RespoDiff
Figure 7: Qualitative comparison of RespoDiff, SD, and SDisc using the prompts "a photo of Analyst" (top) and photo of a Cashier" (bottom).
8.4.6 Evaluation Metrics
We employ the modified deviation ratio, as defined in Li et al. 20024). quantify the fairness of where C is the total number of attributes in a societal group, N is the total number of generated images, and No denotes the number images classified as attribute C. The deviation ratio quantifics attribute disparity. with 0 < A < 1; lower A indicates more balance, while higher A shows greater imbalance. We utilize the CLIP classifier (Radford et al. 202T) to evaluate the generated images by calculating the similarity between each image and relevant prompts, assigning the image to the class with the highest similarity score.
the generated images. The deviation ratio computed as A max.,e
1-
We assess image fidelity using the FID score (Heusel et al. 2017) on the COCO-30k validation set, while image-text alignment is measured with the CLIP score (Radford et al. 202T) using COCO-30k prompts. We also assess the alignment between the generated images and the Winobias prompts AWinoAlign) used to generate them. This metric enables us to evaluate how well the generated images correspond to prompts containing profession-related terms. This evaluation is crucial, as
23

---

any debiasing approach must not only ensure faimess but also maintain alignment with the specified professions.
8.4.7 Quantitative Results for Extended Winobias Dataset
We report the average deviation ratio and prompt-image alignment using WinoBias prompts on the extended WinoBias dataset for both gender and race in Table II and Table 12 respectively. The results demonstrate that our approach achieves more balanced generation and improved text-image alignment compared to existing methods, even in this challenging setting.
Table 11: Fairness (Gender +) with Sta- Table 12: Fairness (Race with Stable
ble Diffusion vl.4
Diffusion v1.4
Approach
Dev Rat (1)
WinoAlign (1)
Approach
DevRat (1)
WinoAlign (f)
SD 0.70
27.16
SD
0.48
27.16
SDisc
0.23
26.61
SDisc
0.20
27.08
FDF
0.39
23.90
FDF
0.24
23.56
BAct
0.60
27.49
BAct
0.39
27.50
FD
0.31
27.50
FD
0.45
27.52
RespoDiff
0.15
27.30
RespoDiff
0.16
27.12
8.4.8 Detailed Fairness Metrics for Individual Professions
In the main text, we presented the average deviation deviation ratio across 36 professions. Here, we offer a detailed comparison of the deviation ratio across other baselines. The results are summarized in Table 13 It is evident that the transformations learned by our approach effectively generalize to previously unseen professions, mitigating gender and racial biases without requiring any training on profession-specific data.
professions between our approach and
8.4.9 Debiasing Intersectional Biases
This section evaluates the effectiivneess of RespoDiff in addressing intersectional biases, specifically gender and racial biases. As highlighted by Gandikota et al.) (2024), the prompt "a Native American person" exhibits strong male bias, with 96% of generated images depicting males, emphasizing the need for joint debiasing approach across multiple attributes for fairer generation.
To assess method's effectiveness in mitigating intersectional biases, we perform quantitative analysis across gender and race, as presented Table [4] The results indicate that our approach achieves better balance between fairness and semantic alignment compared to existing methods. While SDisc attains a lower gender deviation ratio, it does so at the cost of significantly degraded visual quality and image-text alignment, as reflected in higher FID scores. Unlike the Fair Diffusion Framework (FDF) (Shen et al.] 2024), which requires additional training, our approach operates without retraining, instead leveraging pre-learned transformations to achieve superior results. This demonstrates our method's capability to effectively mitigate both individual and compounded biases while preserving image fidelity and text-image alignment.
Effect of Composition Scales. We notice some interference when composing gender and race modules, with gender debiasing being affected more strongly than race. We attribute this to initially applying unit scales to both concepts. When composed, the concept with the larger effective residual can dominate, under-steering the weaker one. To examine this, we varied the per-concept scales during composition (e.g., slightly > for gender and slightly < for race).
As shown in Tables 15 and 16 applying light concept-specific scaling rebalances the composition: gender fairness is restored to near its single-concept level without significantly changing alignment, while race faimess remains stable. We will include qualitative results in the final version. Learning the scaling factors dynamically to compose concepts promising future direction for further improving multi-attribute debiasing.

---

Table 13: Fairnsss evaluation results with deviation ratios across different professions. Lower values indicate better fairness.
Dataset
Gender SDise
Gender+
Race SDisc
Race+
Method
SD
FDF
Ours
SD
SDisc
FDF
Ours
SD
FDF
Ours
SD
SDise
FDF
Ours
Analyst
0.70
0.02
0.22 0.07
0.54
0.02
0.03 0.11
0.82
0.23
0.24
0.13
0.77
0.41
0.18
0.08
Assistant
0.02
0.08
0.08
0.04
0.48
0.10
0.23
0.03
0.38
0.24
0.24
0.06
0.24
0.12
0.24
0.17
Atlendant
0.16
0.14
0.25 0.03
0.78
0.10
0.35
0.03
0.37
0.22
0.39
0.17
0.67
0.13
0.42
0.16
Baker
0.82
0.00
0.37 0.09
0.64
0.12
0.35
0.01
0.83
0.12
0.49
0.15
0.72
0.16
0.43
0.02
CEO
0.92
0.06
0,48
0.19
0.90
0.06
0.30
0.14
0.38
0.22
0.15
0.09
0.31
0.22
0.05
0.32
Carpenter
0.92 0.08
0.60
0.11
1.00
0.66
0.84
0.59
0.91
0.28
0.33
0.16 0.83
0.26
0.08
0.21
Cashier
0.74 0.14
0.29 0.09
0.92
0.42
0.65
0,25
0.45
0.34
0.29
0.23
0.46
0,30
0.32
0.13
Cleaner
0,54 0.00
0.09 0.25
0,30
0.22
0.25
0.12
0.10
0.14
0.31
0.16
0.45
0,26
0,32
0.17
Clerk
0.14
0.00
0.05
0.44
0.46
0.59
0.16
04 0.15
Construct. Worker
1.00 0.80
0.88 0.81 1.00
0.82
0.87
0.52
0.41
0.26
0.25
0.20
0.44
0.25
0.21
0.22
Cook
0.72 0.00
0.19 0.01
0.02
0.16
0.09
0.01
0.56
0.30
0.32
0.14
0.18
0.14
0.22
0.25
Counselor
0,00
0.02
0.16 0.04
0.56
0.12
0.47
0.72
0.16
0.48
0,11
0.36
0.12
0.32
0.16
Designer
0.12 0.12 0.31
0.11
0.21
0.16
0.12
Developer
0.90
0.40
0.51
0.53
0.92
0.58
0,40
0.4
0.41
0.30
0.14
0.13
0.32
0.39
0.15 0.14
Doctor
0.92
0,00
0.65
0.03
0.52
0.00
0.20
0.00
0.92
0.26
0,42
0.13
0.59
0.15
0.33
0.17
Driver
0.90
0.08
0.01
0.12
0.48
0.04
0.08
0.08
0.34
0.16
0.13
0.08
0.25
0.07
0.2
0.05
Farmer
0.16
0.51
0.26
0.29
0:95
0.50
0.48
0.39
0.16
Guard
0.78
0.18
0.79
0.28
0.76
0,20
0.64
0.11
0.20
0.12
0.24
0.31
0.35
0.14
0.25
0.15
Hairdresser
0.92
0.72
0.33
0.35
0.88
0.80
0.67
0.60
0.45
0.42
0.36
0.20
0.38
0.23
0,41
0.13
Housekeeper
0,96
0.66
0.91
0.15
1.00
0.72
0.95
0.19
0.45
0.28
0.26
0.34
0.45
0.34
0.26
0.35
Janitor
0.96 0.18
0.71
0.21
0.94
0.28
0.52
0.12
0.35
0.24
0.2
0.02
0.40
0.07
0.24
0.13
Laborer
1.00
0.12
0.42
0.39
0.98
0.14
0.32
0.33
0.24
0.1
0.53
0.20
0.27
0.47
Lawyer
0.68
0.00
0.25
0.04
0.36
0,10
0.03
0.08
0.64
0.18
0.38
0.05
0.52
0.13
0.16
0.14
Librarian
0.66
0.08
0.31
0.04
0.54
0.06
0.24
0.04
0.85
0.42
0.5
0.21
0.74
0.27
0.27
0.09
Manager
0.46
0.00
0.12 0.05
0.62
0.02
0.29
0.05
0.69
0.24
0.29
0.16
0.41
0.19
0.29 0.16
Mechanic
0.69
0.04
0.28
0.15
0.64
0.14
0.19
0.47
0,27 0.04
Nurse
1.00
0.62
0.71
0.09
0.98
0.46
0.79
0.12
0.76
0.30
0.46
0.12
0.39
0.08
0,27
0.27
Physician
0.78
0.00
0.25
0.03
0.30
0.00
0.03
0.03
0.67
0.18
0.28
0.15
0.46
0.02
0.12
0.12
Receptionist
0.84
0.64
0.44
0.36
0.98
0.80
0.60
0.65
0.88
0.36
0.52
0.19
0.74
0.25
0.32
0.11
Salesperson
0.68
0.00
0.55
0.05
0.54
0.00
0.09
0.07
0.69
0.26
0.38
0.12
0.66
0.36
0,26
0.06
Secretary
0.64
0.36
0.08
0.14
0.92
0.46
0.13
0.37
0.24
0.56
0.25
0.55
0.32
0.42
0.18
Sheriff
1.00
0.08
0.89
0.15
0.98
0.14
0.79
0.09
0.82
0.18
0.35
0.08
0.74
0.27
0.31
0.17
Supervisor
0.64
0.04
0.37
0.03
0.52
0.04
0.51
0.08
0.49
0.14
0.23
0.23
0.45
0.14
0.11
0.14
Tailor
0.56
0.06
0.40
0.13
0.78
0.06
0.43
0.11
0.16
0.10
0.23
0,26
0.14
0.13
0,27
0,20
Teacher
0.30
0.04
0.30
0.03
0.48
0.41
0.51
0.43
0.26
0.21
0.24
0.16
Writer
0.04
0.06
0.28
0.01
0.26
0.06
0.49
0.01
0.86
0.26
0.45
0.15
0.69
0.07
0.26
0.17
Winobias (Avg.)
0.68
0.17
0.40
0.14
0.56
0.23
0.32
0.15
0.70
0.23
0.39
0.16
0.48
0.20
0.24
0.16
Table 14: Intersectional biases for Gender and Race with Stable Diffusion v1.4
Appreach
DevRat (Gender)
DevRat (Race)
WinoAlign (30K)
CLII (30K)
27.16
31.33
SDise
0.15
26
28.43
FDF
0,38
0.31
23:15
15.09
30.43
RespoDiff
0.20
0.14
27.11
14.78
29.98
8.4.10 Extension of RespoDiff' to Transformer-based Architectures
In this section, we evaluate the effectiveness of our approach on transformer-based diffusion archi- tectures like Flux. Due to resource constraints, these experiments were performed using Flux- Flux-mini. Unlike UNet-based diffusion models, Flux employs transformer architectures without an explicit intermediate bottleneck space. Therefore, we applied our modules directly to the text representations. Identifying a similarly interpretable latent space within transformers remains interesting direction for future exploration. We evaluate gender debiasing under this setup, and the results are summarized in Table [1]]
The results show that RespoDiff substantially reduces gender bias while maintaining competitive alignment on Flux. This highlights the versatility of our method beyond UNet-based models. Overall, our work provides extensive evaluations across diverse architectures, going beyond prior responsible generation methods, and supporting the robustness and generality of RespoDiff.
25

---

Table 15: Gender fairness before and after composition with Race.
Setting
Dev Ratio (1)
WinoAlign (1)
Gender only
0.14
27.30
Gender + Race (Scales 1, 1)
0.20
27.12
Gender Race (Scales 1.1. 0.9)
0.15
27.09
Table 16: Race fairness before and after composition with Gender.
Setting
Dev Ratio (1)
WinoAlign (1)
Race only
0.16
27.53
Gender Race (Scales 1, I)
0.14
27.12
Gender Race (Scales 1.1, 0.9)
0.16
27.09
8.4.11 Comparison LoRA-based Approaches
We compare RespoDiff with LoRA-based approaches on the Doctor profession to evaluate plug- and-play alternative for gender debiasing. We trained two concept-specific LoRAs ("male doctor" and "female doctor") using SD v1.4 with the default HuggingFace training arguments (approximately 15k iterations per LoRA). At inference, we prompted "a photo of a doctor," uniformly sampling the two LoRAs and selecting the best fuse scales we found (male=1.0, female=1.2). The results reported in Table
We observe that the LoRA approach yields much higher deviation ratio than RespoDiff, while aligmment remains comparable. This suggests that, despite its simplicity, RespoDiff achieves sig- nificantly better fairness outcomes than LoRA in our setting. Also, compared to LoRA, RespoDiff introduces negligible overhead inference. Our modules are applied only once per denoising step on the bottleneck latent representation, resulting in minimal additional computation. In contrast, LoRA typically modifics multiple cross-attention layers throughout the U-Net, introducing repeated low-rank matrix operations and increasing both memory usage and latency. In this regard, RespoDiff' provides plug-and-play mechanism for responsible generation with significantly lower runtime burden.
8.5
Safe Generation
In this section, we discuss some additional experimental details that we utilize for safe generation experiments.
8.5.1 Evaluation Metrics
To assess inappropriateness in the generation, we utilize a combination of predictions from the Q16 classifier and the NudeNet classifier on the generated images, in line with the approaches presented in Gandikota et al. (2023); Schramowski et al. (2023): : Li et al. (2024). The Q16 classifier determines whether image is inappropriate, while the NudeNet classifier identifies the presence of nudity, An image is categorized as inappropriate if either classifier returns positive prediction. We evaluate the accuracy of the generated images using Q16/Nudenet predictions, which quantify the level of inappropriateness. We generate five images for each prompt the 12P benchmark during the Q1 6/NudeNee accuracy evaluation.
evaluate image fidelity using the FID score (Heusel et al. 2017 on the COCO-30k validation set, while image-text alignment is measured with the CLIP score (Radford et al., 2021) using COCO-30k prompts under the safe concept direction. All results are averaged over two random seeds, and the mean values are reported throughout the experimental section.
26

---

Table 17: Comparison of Flux and RespoDiff (applied to Flux) on gender debiasiing
Approach
Dev Ratio (4)
WinoAlign (1)
Flux RespoDiff 0.27
0.71
24.73
23.65
Table 18: Comparison of LoRA and RespoDiff gender debiasing for the Doctor profession.
Approach DevRat (1) Prompt Alignment (1)
LoRA
0.41
28.09
RispoDiff
0.03
28.15
8.5.2
Baselines
We compare the performance of our proposed approach against three safe generation baselines: (1)
SD (2) ESD (Gandikota et al. 2023), erases concepts by fine-tuning the cross-attention layers (3)
SLD (Schramowski et al. 2023). modifies the inference process to ensure safe generation.
8.5.3
Quantitative Results for Individual I2P Categories
Table 19: Safety results for all 7 categories individually.
Approach
Harassment
Illegal activity
Self-harm
Shocking
Violence
(avg)
SD
0.34 0.02
0.41 1 0.03
0.34 0.02
0.44 0.02
0.38 0.02
0.51 1 0.02
0.44 1 0.02
0.27 0.01
SDise
0,18 = 0.02
0.29 + 0.03
0.23 : 0.02
0.28
0.02
0.22 0.01
0.36 2 0.02
0.30 :: 0.02
0.27 001
SLD
0.15
0,18 0.03
0.17 1 0.02
0.19 0.02
0,15
1 0.01
0.32 1 0.02
0.21 1 0.02
0,20 001
ESD
0.27 0.02
0.32 0.03
0.33 0.02
0.02 0.35
0.18
0.01
0.41 + 0.02
0.41 + 0.02
0.32 0.01
RispoDiff
0.13 0.02
0.15 + 0.01
0.13 0.01
0.16 0.01 0.12 0.02
0.26 + 0.01
0.16 = 0.00 0.16 = 0.01
In the main paper, we reported the average 12P benchmark metrics across seven categories. Ad- ditionally, we present detailed analysis of safety metries for each individual category in the I2P benchmark dataset, as shown in Table [9] Notably, our approach is trained using only anti-sexual and anti-violence concepts. However, the results in Table T9] demonstrate that our method effectively generalizes to unseen categories within the I2P benchmark, highlighting its strong adaptability.
8.5.4 Extension to SDXL
In this section, we present experimental results evaluating RespoDiff with SDXL for safe generation. The results are reported in Table 20
We observe that RespoDiff effectively eliminates inappropriate content while maintaining competitive image fidelity on SDXL, consistent with its performance on SD 1.4.
8.6 Ablation on Different Architectures for Transformations
Table 21 presents an ablation study evaluating different architectural choices for the transformation modules in the context gender debiasing. Our approach applies a constant function that is linearly added to the bottleneck activation, ensuring a minimal yet effective modification to the model's 's latent space. However, as described in Section 4.] we incorporate both RAM and SAM transformations by adding them to the same activation. Naively summing all components would result in the bottleneck activation being added twice, potentially leading to out-of-distribution generations. To avoid this, we ensure the bottleneck activation is added only once, and then added to RAM and SAM transformations.
To assess the impact of different architectures, we also experiment with MLP-based transformations and three convolutional (Conv) layers. The results indicate that our method achieves the lowest deviation ratio while maintaining competitive semantic alignment to the prompts. The MI.--based transformation and convolutional-basee transformations yield a slightly higher deviation ratio but comparable alignment performance. These findings highlight that our approach is invariant to the
27

---

Table 20: Comparison of RespoDiff with SDXL on 12P, FID, and CLIP metrics.
Approach
I2P ())
FID (30K) (I)
CLIP (30K) (1)
SDXL 0.34
13.68
32.19
RispoDiff
0.17
13.90
32.10
Table 21: Ablation on modules with different architectures (Gender debiasing).
Module
DevRat ())
WinoAAlinn (1)
MLP Conv
0.17
27.
0.16
27.51
RespoDiff
0.14
27.30
architecture used for the transformations. However, using constant function added to the bottleneck vectors better balances fairness and semantic alignment compared more complex architectures. In short, we present RAM and SAM general transformation modules to keep the framework flexible and extensible. RispoDiff's contributions do not rely on any specific parameterization. RAM and SAM can be replaced with richer, dependent modules if desired, and ablations confirm the method remains effective under such changes.
8.7 Robustness to Alternate Neutral Prompts
For fairness in human subjects, we adopt the neutral prompt "a person," which we find generalizes well across diverse human contexts. For safety, we use "a scene," as it captures broad range of environments and settings. As shown in the main results, both prompts effectively support generalization unseen scenarios. To further assess the robustness of our approach to the choice of neutral prompt, we conduct an additional experiment. We retrained the man and woman modules using an alternative neutral prompt "a group of people," and evaluated them on two out-o--distributton prompts: "a group of friends on picnic" and "a couple of doctors." " This allows us to test whether our method remains effective when trained on more complex neutral formulations. Due to the limitations of CLIP in capturing fairness and alignment for complex prompts, employed GPT-4o for evaluation. We generated 200 images, uniformly sampling male and female modules, and utilised GPT-4o to: (i) classify each image male female (to compute deviation ratio), and (ii) rate alignment to the prompt on a 1-5 scale, with images scoring 2 4 considered well-aligned. The results summarized in Table 22
Table 22: Evaluation of RespoDiff trained with the neutral prompt "a group of people" on out-of- distribution prompts.
Prompt
Deviation Ratio ())
Alignment Accuracy (% 1)
A group of friends on a picnic A couple of doctors
0.026
96.5
0.029
95.1
Our results indicate that training with different complex neutral prompt also yields balanced gender representations on arbitrary unseen queries such as "a group of friends on a picnic," while preserving prompt alignment. This suggests that the approach generalizes to other generic neutral prompts.
8.8 Similarity-base Strategy for Automatically Identifying Neutral Prompts
For real-world deployment of RespoDiff, it important to automatically identify neutral prompts rather than relying on manually predefined ones. To this end, we envision first generating small set of candidate neutral prompts using LLMs, and then training RespoDiff modules on this pool. At inference, the system can select the most suitable module by measuring similarity between the user prompt and the candidate neutral prompts. For example, a prompt such as "a group of friends on picnic" would naturally align with module trained on "a group of people".
28

---

To evaluate this idea, we implemented a similarity- based strategy for neutral prompt selection at inference time. Specifically, we used CLIP's text encoder compute cosine similarities between user prompts and small set of general-purpose candidate neutral prompts (e.g., "a person", "human", group of people"). We considered both profession-bant from WinoBias and group-based prompts such as "A couple of doctors" and "A group of friends on picnic" as test cases. Results for subset of prompts are reported in Table 23
prompts (e.g., photo of a doctor")
Table 23: Cosine similarity between user prompts and candidate neutral prompts using CLIP.
Target Prompt
a person
human
a group of people
A photo of an Analyst
0.8682
0.8215
0.8068
A photo of an Assistant
0.8753
0.8352
0.8423
A photo of an Attendant
0.8461
0.8086
0.7759
A photo of a Baker
0.8281
0.7988
0.7854
A photo of a CEO
0,8731
0.8532
0,8387
A photo of a Carpenter
0.8321
0.7952
0.7795
A photo of a Cashier
0.7791
0.7426
0.7090
photo of a Cleaner
0.8439
0.8226
0.7950
A group of friends on a picnic
0.7739
0.7361
0,8725
couple of doctors
0.8128
0.8023
0.8375
Across all 36 profession prompts from WinoBias, the closest match was consistently "a person", while group-based prompts aligned most strongly with "a group of people". This indicates that the automated discovery method naturally recovers the same neutral prompts used during training. Since RespoDiff has already been shown to perform effectively with these prompts in both our main experiments ("a person") and the additional evaluations with alternate neutral prompt ("a group of people"), these findings provide preliminary empirical support for the inference-time time prompt-matching strategy.
8.9 Qualitative Results
In this section, we present supplementary qualitative analyses for all tasks discussed in the main text. Fig. 9 and Fig. 10 provide further quantitative evaluations of gender- and race debiasing for other professions using our approach. Additionally, Fig. present qualitative analyses demonstrating the effectiveness of safety transformations in reducing harmful content generation. We also present qualitative results that show the effectiveness of integrating our approach to SDXL in Fig. and Fig. 12 respectively.
Figure 8: Qualitative comparison of safe generation. RespoDiff removes nudity and violence, compared to SD.
29

---

Stabie Diftaiion
Oue-
Woeue
Suble Miliio
Out
Stuble Diflasio
Cure-
Oue
Suile
Oui
Figure 9: Comparison of RespoDiff and SD in generating images with respect to gender (woman, man) across various professions, Nurse, Firefighter, Cook and Analyst. RespoDiff ensures responsible representation while preserving fidelity to the original SD outputs.
30

---

Subk Dilluskon
Own
Onn
Stable Dificiin
Diftaiio
Figure 10: Comparison of RespoDiff and SD in generating images with respect race (Black, Asian, White) across various professions, CEO, Construction worker, Cashier. RespoDiff ensures responsible representation while preserving fidelity the original SD outputs.
31

---

SDXL
SDXL
BINI
SDXL
Figure 11: Comparison of RespoDiff and SDXL in generating images with respect to gender (Woman, Man) across various professions, Baker, Lawyer, Cook and Sheriff. RespoDiff ensures responsible representation while preserving fidelity to the original SDXL outputs.
32

---

Mook
Figure 12: Comparison of RespoDiff and SDXL in generating images with respect to race (Black, Asian, White) across various professions, Doctor, Cook and Salesperson. RespoDiff ensures respon- sible representation while preserving fidelity to the original SDXL outputs.
33


Reviews:
[{"review_id": "ZVkgODyFyi", "reviewer": "Reviewer_xWgu", "summary": "This paper proposes RespoDiff, a dual-module transformation framework for improving the fairness and safety of text-to-image diffusion models. One module (RAM) steers generations toward responsible concepts such as demographic diversity or safe content, while the other module (SAM) aims to preserve consistency with neutral prompts. The paper introduces a score-matching objective to coordinate both modules and demonstrate improved results on fairness and safety benchmarks using both Stable Diffusion and SDXL.", "questions": "1.\tCould you include comparisons to lightweight debiasing or editing methods, such as LoRA or adapter-based fine-tuning? This would help contextualize the added cost and complexity of your proposed method.\n2.\tHow did you choose the training schedule (e.g., 1500 iterations with batch size 1), and did you observe stability or overfitting issues? Would larger batch sizes or early stopping change the results?\n3.\tCan you provide more comprehensive visual evaluation, such as class-wise breakdowns, distributional shifts, or qualitative failure cases, to complement the current figures?\n4.\tAre there strategies to improve FID when applying responsible transformations, or is some degradation unavoidable?\n5.\tWhat is your view on extending this method to support unseen or emergent concepts not included in the original training categories?", "limitations": "No. The paper would benefit from a more direct discussion of its limitations. Specifically, the reliance on predefined concept sets and separate training per category introduces scalability and coverage concerns. Additionally, the societal impact of applying targeted transformations (e.g., selecting what constitutes a “safe” or “fair” output) should be considered more carefully, as these decisions may encode implicit normative judgments.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Addressing bias and unsafe outputs in generative models is a critical challenge. The paper focuses on a practical and socially relevant goal: producing images that better reflect demographic balance and remove harmful content, without severely degrading fidelity. This target is timely given ongoing concerns with fairness in large-scale image generation.\n2.\tSplitting the model’s behavior into two learnable modules, target concept steering + preserving alignment, is a clean and modular approach. This separation makes the optimization more interpretable and offers a more principled alternative to ad hoc fine-tuning or prompt hacking. The score-matching formulation used to couple these modules also aligns with how diffusion models are typically optimized.\n3.\tRespoDiff outperforms existing methods such as SDisc, FDF, and BAct on deviation ratio and safety metrics across multiple datasets. For example, on the I2P benchmark, the model reduces the rate of unsafe generations significantly compared to SD and SLD, while maintaining competitive CLIP scores and acceptable image fidelity.\n\nWeaknesses:\n1. The paper briefly evaluates the individual contributions of RAM and SAM (Table 6), but stops short of deeply analyzing alternative designs or loss settings. For example, it would be helpful to evaluate whether a shared transformation performs substantially worse, or how sensitive the model is to the λ parameter balancing the two objectives. \n2.\tEach concept category (e.g., gender, race, safety) requires training transformations using thousands of steps. For instance, the safety module uses 1500 iterations with batch size 1. This fine-grained per-category training increases implementation overhead and raises questions about scalability. It would strengthen the work to compare with plug-and-play alternatives, such as LoRA or prompt reweighting, to show that the added complexity provides significant gains.\n3.\tThe paper presents only a few examples (e.g., Fig. 2 and 3) to support visual performance claims. While illustrative, these are not enough to assess how consistently the method performs across prompt diversity. Additional aggregated qualitative results (e.g., distribution analysis, randomly sampled generations, user preference studies, or error analysis) would better support the fidelity and alignment claims.\n4.\tAlthough RespoDiff maintains semantic consistency and improves fairness, it introduces a drop in image fidelity, especially in the safety task (Table 3). For example, FID increases from 14.09 to 17.89 on COCO-30K when responsible transformations are applied. This suggests that there is still a trade-off between fairness and image quality, and it would be helpful to better understand or mitigate this.\n5.\tThe model requires target concepts to be known ahead of time and does not generalize to new or intersectional identities outside of those categories (e.g., non-binary, mixed-race, or transgender). This predefined concept reliance is acknowledged briefly in the appendix, but not sufficiently addressed in the main paper. Broader or adaptive concept discovery would be needed to make the approach more flexible in practice.", "was_revised": true, "final_justification": "The authors effectively addressed the concerns from me (sensitivity of $\\lambda$, and LoRA comparison, etc) and other reviewers, so I decide to raise the score.", "rebuttal": "We thank the reviewer for the valuable feedback and thoughtful comments. We address each of the concerns below:\n \n\n### **W1/Q1. Shared Transformation and Sensitivity to $\\lambda$**\n\n**Shared Transformation:**  We would like to clarify that the shared transformation design **has been evaluated in our supplementary material** (Section 7.7; Table 15). Specifically, training a single transformation module using the combined loss underperforms compared to our proposed two-module setup (RAM + SAM) across all metrics. We believe this is because concept steering and semantic alignment are orthogonal objectives. When both are optimized through a single shared transformation, their competing gradients interfere, preventing effective specialization and resulting in suboptimal performance on both fairness and fidelity.  In contrast, our modular design separates responsibilities: RAM applies strong, concept-specific transformations, while SAM maintains semantic consistency by remaining close to the identity. This decoupling enables each module to specialize effectively, resulting in improved performance across both dimensions. Additionally, such control is especially valuable for real-world deployment and interpretability, where the ability to tune each component independently is highly desirable. Achieving this level of modularity and transparency is challenging with a shared transformation approach.\n\n**Sensitivity to  $\\lambda$:** We now present results on the sensitivity of RespoDiff to the hyperparameter $\\lambda$.\n\n| λ  | Dev Ratio (↓) | WinoAlign (↑) | FID (30K) ↓ | CLIP(30K) ↑ |\n|------------|--------------|------------------|------------------|-------------------------|\n| 0          | 0.12         | 26.12            | 15.63            | 29.93                   |\n| 0.5 (Default) | 0.14      | 27.30            | 14.91            | 30.67                   |\n| 4          | 0.29         | 27.53            | 14.17            | 31.24                   |\n\n\nEmpirically, increasing $\\lambda$ keeps the trajectory closer to the base model, improving fidelity but weakening target‑concept steering, while decreasing $\\lambda$ has the opposite effect. We therefore use $\\lambda =0.5$ as a knee point that balances fairness and fidelity.\n\n### **W2. Comparison to LoRA**\n\nWe now provide comparisons to LoRA-based approaches on the Doctor profession to test a plug‑and‑play alternative for gender debiasing. We trained two concept‑specific LoRAs (“male doctor” and “female doctor”) using SD v1.4 with the default HuggingFace training arguments (≈15k iterations per LoRA). At inference, we prompted “a photo of a doctor”, uniformly sampling the two LoRAs and selecting the best fuse scales we found (male=1.0, female=1.2).\n\n| Approach   | DevRat ↓ | Prompt Alignment ↑ |\n|------------|----------|---------------------|\n| LoRA       | 0.41     | 28.09               |\n| RespoDiff  | 0.03     | 28.15               |\n\nIt can be observed that the LoRA approach produced a much higher deviation ratio than RespoDiff, while alignment was comparable. This suggests that, despite its simplicity, RespoDiff achieves significantly better fairness outcomes than LoRA in our setting.\n\n**Scalability:** While RespoDiff requires training (~1,500 steps) per concept per neutral prompt, this cost is incurred only once and is not tied to specific user prompts. For instance, modules trained using general prompts like “a person” (for fairness) or “a scene” (for safety) generalize effectively to diverse, unseen prompts without further finetuning, as demonstrated in our profession and I2P evaluations. Thus, we believe that the per‑concept cost is amortized over a wide prompt distribution.\n\n**Inference Overhead:** Compared to LoRA, RespoDiff introduces negligible overhead at inference. Our modules are applied only once per denoising step on the bottleneck latent representation, resulting in minimal additional computation. In contrast, LoRA typically modifies multiple cross-attention layers throughout the U-Net, which introduces repeated low-rank matrix operations and increases both memory and latency. In this regard, RespoDiff offers a plug-and-play mechanism for responsible generation with significantly lower runtime burden.\n\n### **Q2. Choice of training schedule** \n\n\nWe chose the schedule based on loss curves and qualitative analysis. Both the concept and semantic losses converge, and visualisations stabilize by ~1,500 iterations; training much longer (e.g., ≥10,000 iterations) introduced visible artifacts and overfitting to certain visual types. Larger batch sizes slightly increased the concept loss, but the qualitative results were essentially unchanged. Accordingly, we adopt 1,500 iterations with a small batch, which yields stable visuals with converged losses.\n\n### **W3/Q3. Qualitative results**\n\n Beyond Figs. 2–3 in the main paper, we provide extensive qualitative results in the supplementary: Sec. 7.8 (Figs. 8–9) presents SD v1.4 generations across a wide range of professions, Figs. 10–11 present corresponding results for SDXL, and Fig. 7 shows qualitative examples for safe generation. Across these diverse prompts, we believe RespoDiff reliably steers toward the intended target concept while maintaining visual fidelity. While we are unable to include additional qualitative analyses in the rebuttal phase due to the restriction on image submission, we appreciate the reviewer’s feedback and plan to incorporate further qualitative results, such as failure cases in the final version of the paper.\n\n### **W4/Q4. FID for safety**\n\nWe agree there is a trade‑off when enforcing safety. We believe this arises from the nature of safety‑oriented transformations and how FID is computed. Safety controls suppress potentially unsafe visual cues (e.g., weapon‑like shapes, exposed‑skin/texture patterns), even when such cues are faint or unintentional. Because these cues are entangled with broader scene/style attributes (lighting, background textures), the transformations can introduce distortions but systematic changes (extra coverage, local occlusions, background adjustments). These shifts move the generated distribution away from COCO’s statistics, which FID, measured in Inception feature space penalizes [A], even when semantic alignment remains high (consistent with our CLIP results in Table 3). By contrast, we believe that fairness‑oriented edits are typically localized identity adjustments (skin tone, facial cues) that preserve scene layout, lighting, and background statistics, so the induced distribution shift and thus the FID change is much smaller.  This behavior aligns with findings by Jayasumana et al. [B], where they show that FID increases with high distortions while being less sensitive to low distortions. Even with semantic regularization, some safety cases unavoidably move farther from the base distribution to remove unsafe cues, while keeping semantics intact. Within these constraints, our results indicate that RespoDiff improves safety more than prior methods while avoiding large fidelity degradation, yielding a comparatively stronger safety fidelity balance.\n\n[A] Jung et al. Internalized Biases in Fréchet Inception Distance NeurIPS 2021, DistShift Workshop\n\n[B] Jayasumana et al., Rethinking FID: Towards a Better Evaluation Metric for Image Generation, CVPR, 2024.\n\n### **W5/Q5. Extension of RespoDiff to unseen concept categories/concept discovery**\n\nWe appreciate this point. We view responsible generation as comprising two complementary stages: (i) identifying the relevant fairness/safety concepts, and (ii) mitigating bias or unsafe content given those concepts. Our work addresses **(ii) mitigation**. Consistent with prior mitigation approaches, we assume the fairness/safety concepts are known in advance, which we acknowledged in the limitations. To the best of our knowledge, this is a common assumption in prior fair/safe generation works  (e.g., Li et al., 2024; Gandikota et al., 2024; Shen et al., 2024; Chuang et al., 2023). As noted in the main paper (Line 152), our categories and concepts are chosen to align with these works. That said, we believe that extending to unseen or emergent concepts is practical within the same framework since RespoDiff is modular. Each concept corresponds to a lightweight transformation. Hence, for any new concept, we can train an additional module without changing the backbone. Moreover, our approach can mitigate intersectional biases without additional training by composing existing transformations, as demonstrated in Section 7.4.9, which covers unseen combinations of known concepts and thus partially addresses “unseen” identities. However, we also agree that unseen/emergent concept discovery is important in practice and is orthogonal to our contribution. One approach would be to utilise a world‑model or LLM‑based procedure to propose new or emergent identities [C], after which RespoDiff can learn the corresponding transformations straightforwardly. We will make these points explicit in the main paper. \n\n[C]  D’Incà et al.,OpenBias: Open-set Bias Detection in Text-to-Image Generative Models, CVPR 2024.\n\nWe thank the reviewer once again for their valuable feedback and hope that our responses have addressed all the concerns raised."}, {"review_id": "M7LAPf7IlE", "reviewer": "Reviewer_oss3", "summary": "This paper proposes a framework for responsible and faithful text-to-image generation via dual-module bottleneck transformation in diffusion models. The method targets two critical yet often conflicting objectives: enhancing fairness/safety (e.g., reducing gender or racial bias, eliminating unsafe content) while preserving semantic fidelity to the input prompt. RespoDiff operates in the bottleneck latent space of a pre-trained diffusion model, introducing two learnable modules: Responsible Concept Alignment Module (RAM): steers generation toward fairness-aligned or safety-oriented concepts. Semantic Alignment Module (SAM): regularizes the transformation to maintain alignment with the original prompt's semantics.", "questions": "1. The training process assumes access to a well-defined neutral prompt. In practice, this may be subjective or culturally sensitive. Could the authors provide a strategy for automatically identifying or validating neutral prompts?\n\n2. Can RespoDiff handle prompts that contain multiple overlapping responsible concepts (e.g., “a young Asian female doctor”)? Is there interference when applying multiple transformations, and if so, how is that managed?\n\n3. Have the authors evaluated the method under ambiguous or adversarial prompts? How robust is RespoDiff to prompts that implicitly invoke bias?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "RespoDiff's strength lies in the modular design that decouples ethical alignment from semantic preservation, enabling interpretable and reusable concept transformations. \nHowever, two weaknesses remain. 1.RespoDiff relies on predefined, binary demographic concepts (e.g., \"man\" vs. \"woman\", or \"Black\" vs. \"White\"), limiting representation of non-binary concepts. 2, robustness to adversarial prompts or domain shifts is not explored. 3. the method assumes the availability of a neutral prompt, which may not always be easy to define in practice.", "was_revised": true, "final_justification": "The rebuttal and additional experiments provide strong empirical support for the method’s robustness and generality in the tested settings. While some opportunities remain for extending cultural/linguistic coverage, the key methodological concerns have been resolved to my satisfaction. Given these improvements, I am inclined to raise my score.", "rebuttal": "We thank the reviewer for the valuable feedback and thoughtful comments. We address each of the concerns below:\n\n### **W1. Reliance on predefined, binary demographic concepts**\n\nOur categories are not strictly binary - for race, consistent with prior works (Li et al., 2024), **we adopt three categories** (Black, Asian, White) as noted in Line 150 of the main paper. For gender, we currently consider man/woman; we acknowledge that this does not capture non‑binary identities and **explicitly note this limitation in Section 7.1**. We chose these categories to align with prior works (Li et al. 2024) and ensure a fair comparison.\nRegarding predefined concepts, we view responsible generation as comprising two complementary stages: (i) identifying the relevant fairness/safety concepts, and (ii) mitigating bias or unsafe content given those concepts. **Our work addresses (ii) mitigation**. Consistent with prior mitigation approaches, we assume the fairness/safety concepts are known in advance, which we acknowledged in the limitations. To the best of our knowledge, this is a common assumption in prior fair/safe generation works  (e.g., Li et al., 2024; Gandikota et al., 2024; Shen et al., 2024; Chuang et al., 2023). As noted in the main paper (Line 152), our categories and concepts are chosen to align with these works. That said, we believe that extending to unseen or emergent concepts is practical within the same framework since RespoDiff is modular. Each concept corresponds to a lightweight transformation. Hence, for any new concept, we can train an additional module without changing the backbone. Moreover, our approach can mitigate intersectional biases without additional training by composing existing transformations, as demonstrated in Section 7.4.9, which covers unseen combinations of known concepts and thus partially addresses “unseen” identities. However, we also agree that unseen/emergent concept discovery is important in practice and is orthogonal to our contribution. One approach would be to utilise a world‑model or LLM‑based procedure to propose new or emergent identities [C], after which RespoDiff can learn the corresponding transformations straightforwardly. We will make these points explicit in the main paper.\n\n[C]  D’Incà et al., OpenBias: Open-set Bias Detection in Text-to-Image Generative Models, CVPR 2024.\n\n### **W2/Q3. Robustness to ambiguous prompts or domain shifts**\n\n**Regarding domain shifts:** Our modules are trained using general-purpose neutral prompts (“a person” for fairness, “a scene” for safety), but are evaluated on a wide variety of unseen prompts without any additional fine-tuning. For fairness, this includes profession-related prompts, while for safety, we use the I2P benchmark consisting of real-world, safety-critical queries, introducing a clear prompt distribution shift. Additionally, we assess image fidelity and alignment on the COCO-30K dataset, which spans diverse, non-human-centric domains. Across these varied settings, RespoDiff consistently improves fairness and safety metrics while preserving image quality (see Tables 1–3), demonstrating strong robustness to domain shifts. Moreover, our fairness evaluation is intentionally human-centric, as attributes such as gender and race are specifically defined for human domains. \n\n**Regarding ambiguous prompts that implicitly invoke bias:** Yes, we explicitly evaluate RespoDiff on ambiguous prompts that implicitly invoke bias, **as detailed in Section 7.4.7 of the supplementary**. Specifically, we consider prompts such as \"a photo of a successful doctor\" or \"an image of a successful teacher,\" since terms like \"successful\" are known to amplify implicit gender biases in generation (Gandikota et al., 2024). Section 7.4.1 provides a detailed discussion on our prompt construction process. As shown in the results in Section 7.4.7, RespoDiff effectively mitigates such biases without degrading semantic fidelity, even in these challenging prompt scenarios.\nAdditionally, we emphasize that our default evaluation setting, which uses professions from the WinoBias set (e.g., \"doctor,\" \"CEO\"), already tests the model's robustness to implicitly biased prompts. Such prompts commonly trigger stereotypical gendered images (e.g., predominantly male for \"CEO\" or \"doctor\"), and addressing these implicit biases is precisely the focus of our mitigation approach\n\n### **W3/Q1. Assumption on the availability of a neutral prompt**\nFor fairness in human subjects, we adopt the neutral prompt “a person,” which we find generalizes well across diverse human contexts. For safety, we use “a scene,” as it captures a broad range of environments and settings. As shown in Tables 1–3, both prompts effectively support generalization to unseen scenarios. To further assess the robustness of our approach to the choice of neutral prompt, and in response to the reviewer’s feedback, we conducted an additional experiment. We retrained the man and woman modules using an alternative neutral prompt “a group of people”, and evaluated them on two out-of-distribution prompts: “a group of friends on a picnic” and “a couple of doctors.” This allows us to test whether our method remains effective when trained on more complex neutral formulations. Due to the limitations of CLIP in capturing fairness and alignment for complex prompts, we employed GPT-4o for evaluation. We generated 200 images, uniformly sampling male and female modules, and utilised GPT-4o to:  \n\n- Classify each image as male or female (to compute a deviation ratio).  \n\n- Rate alignment to the prompt on a 1–5 scale. Images scoring ≥4 were considered well-aligned.\n\n| Prompt                         | Deviation Ratio (↓) | Alignment Accuracy (↑) |\n|-------------------------------|----------------------|-------------------------|\n| A group of friends in the picnic | 0.026               | 96.5%                  |\n| A couple of doctors             | 0.029               | 95.1%                  |\n\nOur results indicate that training with a different complex neutral prompt also yields balanced gender representations on arbitrary unseen queries such as ‘A group of friends in the picnic’,  while preserving prompt alignment, suggesting that the approach generalizes to other generic neutral prompts. We will present the qualitative results that support our analysis in the final version of our paper.\n\n**Strategy for automatically identifying or validating neutral prompts:** As demonstrated in our experiments, our approach generalizes well across unseen contexts using a small set of general-purpose neutral prompts (e.g., “a person,” “a group of people”). To identify the neutral prompts, we envision training RespoDiff modules on a limited set of such prompts. At inference time, the system can select the most appropriate trained modules by computing similarity between the arbitrary user prompt and the available neutral prompts. For instance, a user prompt like “a group of friends on a picnic” would naturally align with a module trained on “a group of people.” \n\n\n### **Q2. Handling prompts that contain multiple overlapping responsible concepts**\n\nRespoDiff can support prompts involving multiple responsible concepts by composing the corresponding modules at inference, without requiring additional training, **as detailed in Section 7.4.9**. In our intersectional experiments (gender and race), we noticed some interference affecting gender more significantly than race. We believe this arises because we initially apply unit scales to both concepts. When composed, the concept with the larger effective residual can dominate, under‑steering the weaker one. To probe this, we varied per‑concept steering scales during composition (e.g., slightly >1 for gender and slightly <1 for race).\n\n**Evaluation of Gender fairness before and after Gender, Race composition**\n\n| Setting                          | Dev Ratio (↓) | WinoAlign (↑) |\n|----------------------------------|-----------|----------------|\n| Gender only                      | 0.14      | 27.30          |\n| Gender + Race (Scales 1, 1)      | 0.20      | 27.12          |\n| Gender + Race (Scales 1.1, 0.9)  | 0.15      | 27.09          |\n\n**Evaluation of Race fairness before and after Gender, Race composition**\n\n| Setting                          | Dev Ratio (↓) | WinoAlign (↑) |\n|----------------------------------|-----------|----------------|\n| Race only                        | 0.16      | 27.53          |\n| Gender + Race (Scales 1, 1)      | 0.14      | 27.12          |\n| Gender + Race (Scales 1.1, 0.9)  | 0.16      | 27.09          |\n\nAs shown in the tables, light concept‑specific scaling rebalances the composition, restoring gender steering to near its single‑concept behavior without significantly changing alignment, while race fairness remains stable. We will include these results and qualitative examples in the final version. Additionally, learning the scaling factors dynamically to compose concepts is an interesting direction to explore in future work.\n\nWe thank the reviewer once again for their valuable feedback and hope that our responses have addressed all the concerns raised."}, {"review_id": "8q1RIsBvpb", "reviewer": "Reviewer_eyeL", "summary": "The authors address the problem of responsible image generation, focusing on aspects such as gender, race, and safety. Their approach introduces two modules (RAM and SAM) which are activated after an initial t steps of generation using a neutral prompt. After this phase, training proceeds by alternating between the two modules: RAM is trained to steer the generation trajectory toward a more responsible prompt, while SAM is trained to preserve the content of the original prompt. The outputs of both modules are then combined.\n\nThe method is evaluated on Stable Diffusion 1.4 (SD1.4) and compared to existing techniques using metrics such as average deviation ratio (to assess fairness), prompt alignment, and image fidelity. An additional experiment was conducted on SDXL to test generalization beyond the baseline. An ablation study demonstrates the necessity of combining both RAM and SAM, rather than RAM alone.", "questions": "Q1) Would this work on a non-U-Net architecture, such as FLUX? This is the direction LDMs are moving.\n\nQ2) Why is there no regularization to keep RAM small? If it is not a small offset from SAM, then why couldn't they just be the same?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nS1) The results in Tables 1 and 2 seem reasonable, on SD1.4.\n\nS2) Table 4 shows promising preliminary results, suggesting that the method may also be effective on SDXL.\n\nWeaknesses:\n\nW1) The paper lacks a discussion of related work on diffusion models, which is essential for grounding the method within existing literature.\n\nW2) Figure 1 is highly confusing. Labeling in the latent representations is inconsistent, and the transition from the initial t timesteps to later steps is unclear. The trapezoid shapes used for RAM and SAM suggest dimensionality reduction—if this is intended, it should be clarified. The arrows imply concatenation, yet the accompanying equations indicate element-wise addition. The red arrows for backpropagation are not intuitive, and the figure does not clearly depict the inference-time process.\n\nW3) The term \"SAM\" is already widely used in the literature (e.g., Segment Anything Model), and its reuse here introduces unnecessary ambiguity.\n\nW4) The choice of t, the step at which training shifts from the neutral prompt to RAM/SAM intervention, is neither justified nor explained.\n\nW5) Sections 4.2 and 4.3 are mathematically dense and difficult to follow. The heavy use of in-line equations and nested symbols hinders readability and comprehension.\n\nW6) The primary experiments are conducted on SD1.4, an outdated model with significantly lower visual quality compared to modern diffusion models. This raises concerns about the relevance and practical value of the results. The absence of core experiments on stronger models such as SDXL or FLUX is a major limitation.\n\nW7)Tables 1 and 2 are poorly presented. The key takeaway (that fairness improves with minimal sacrifice to prompt alignment and image quality) is buried in the text. Without careful reading, the results may be misinterpreted as weak.\n\nW8) The ablation study is incomplete. Specifically:\n\nW8a) There is no analysis of the impact of varying t, the intervention start point.\n\nW8b) It is unclear why RAM and SAM are trained in alternating fashion, rather than jointly. A comparison would strengthen the argument for this design choice.\n\nW9) Between the method section and main figure, I found the method overall difficult to understand.", "was_revised": true, "final_justification": "Initially, my main concern was that the experimental models did not adequately reflect the current state of the field, which made it challenging to situate the work within contemporary literature and assess its contribution. However, the authors have now presented convincing results that better establish the relevance of their approach.\n\nThey have also clarified key aspects of their methodology and agreed to revisions that, in my view, significantly enhance the clarity of the manuscript.\n\nWith these primary concerns addressed, I now believe this paper presents a novel and effective approach to an important and timely topic.", "rebuttal": "We thank the reviewer for the constructive and thoughtful feedback. We will carefully revise the manuscript to address all writing-related concerns and improve overall clarity in the final version. Below, we respond to the remaining points in detail.\n\n### **W1. Discussion of related work on diffusion models**\n\nOur primary focus is on enabling responsible (fair and safe) text-to-image generation rather than proposing changes to the diffusion process or diffusion models themselves.  Accordingly, we believe Sec. 3 of the main paper provides sufficient background on diffusion models and includes the most relevant related work to support a clear understanding of our approach. However, if the reviewer feels any important references have been overlooked, we greatly appreciate the suggestions and will be happy to include them in the final version.\n\n### **W2. Regarding Figure 1**\n\nWe clarify that the trapezoid shapes for RAM and SAM were chosen only to visually indicate transformations within the modules, not dimensionality reduction. Similarly, the arrows and red arrows were intended to simplify the visualization and avoid excessive complexity in the figure. We agree, however, that the current labeling can cause confusion and we will address these labeling inconsistencies clearly in the final version of the paper. Also, we will add a figure that depicts the inference-time process in the final version.\n\n### **W4/W8a. The choice of $t$**\nWe believe there may be a misunderstanding regarding RespoDiff's training process. In each training iteration, we sample a timestep $t$ randomly and update the shared RAM and SAM modules using a score-matching objective. These modules are not learned per timestep; instead, they are trained to capture average statistics across the entire diffusion trajectory. This random $t$ training choice is stated explicitly in the paper (lines 174 and 201). Accordingly, **the training intervention point is a randomly sampled $t$ at each iteration**. At inference, the same learned RespoDiff modules are applied at every sampling step to ensure fair and safe generation.  To further address the reviewer's concern regarding the choice of $t$, we conducted **targeted ablations by restricting the training-time sampling range**. Specifically, instead of sampling $t$ from all 1000 DDPM steps, we trained by sampling $t$ under two settings :\n1. $t \\in [600, 1000]$, representing noisier diffusion steps  \n2. $t \\in [0, 350]$ , representing cleaner diffusion steps.  \n\nThe results of these controlled experiments are reported below.\n\n| Range of $t$         | Dev. ratio (↓) | WinoAlign (↑) |\n|---------------------|--------------:|------------------:|\n| 600–1000            | 0.08          | 26.12             |\n| 0–350               | 0.24          | 27.48             |\n| **0–1000 (default)**| 0.14          | 27.30             |\n\nIt can be observed that the noisier steps $[600, 1000]$ reduce deviation to a large extent but severely hurt alignment; cleaner steps improve alignment but hurt fairness. Sampling across the full range offers the most balanced trade-off and is used in our main experiments. We will include this in the final version of our paper.\n\n### **W6/Q1. Relevance of results on SD1.4 and adaptation to Flux**\n\n**Relevance of results on SD1.4 and additional safety results on SDXL :** We conducted the majority of our experiments and comparisons using Stable Diffusion v1.4 (SD v1.4), primarily because **prior responsible-generation methods (Li et al. 2024) have predominantly used SD v1-based models**. To ensure fair and meaningful comparisons, we adhered to the same setup. However, we recognize the importance of evaluating our approach on newer models. Therefore, **we provided core fairness results on SDXL as well**, reported in Table 4 and Table 5 of the main paper. We also provide qualitative analysis on SDXL in Figure 4, 10 and 11 to support our quantitative results. Since the existing baseline methods do not yet support SDXL, we compared our approach against the default SDXL baseline. Our results clearly demonstrate that RespoDiff effectively mitigates gender and racial biases in SDXL without compromising image fidelity. Furthermore, to substantiate our claims regarding the capability of RespoDiff to handle responsible generation using SDXL, we have now also conducted safety experiments on SDXL and present the corresponding results below. From the table, it can be observed that RespoDiff successfully eliminates inappropriate content while maintaining competitive image fidelity on SDXL. \n\n| Approach   | I2P (↓) | FID(30K) (↓) | COCO(30K) (↑) |\n|------------|---------|------------|-------------|\n| SDXL       | 0.34    | 13.68      | 32.19       |\n| RespoDiff  | 0.17    | 13.90      | 32.10       |  \n\n**Adaptation to Flux:** Following the reviewer's suggestion, we conducted additional experiments to investigate the effectiveness of our approach on Flux. Due to resource and time constraints, we performed these experiments with Flux-mini. Since Flux uses transformer architectures without an explicit intermediate bottleneck space (as opposed to UNet architectures), we applied our modules directly to the text representations. Identifying a similarly interpretable latent space within transformers as in UNet, required further exploration. We evaluated gender debiasing using this setup, and the results are summarized below.\n\n| Approach   | Dev Ratio (↓) | WinoAlign (↑) |\n|------------|----------------|---------------------|\n| Flux       | 0.71           | 24.73               |\n| RespoDiff  | 0.27           | 23.65               |\n\nThe experimental results demonstrate that our approach can effectively enable responsible generation even on architectures like Flux. We will also provide the qualitative results for Flux in the final version of our paper. We believe this highlights the versatility of RespoDiff beyond UNet-based models. Overall, our work includes extensive evaluations across diverse model architectures, going beyond prior responsible generation methods, to support the robustness and generality of our approach.\n\n### **W8b. Joint training of RAM and SAM**\n\nDuring experimentation, we found that jointly training RAM and SAM using our combined loss led to higher deviation ratios and weaker alignment compared to our proposed alternating strategy. We believe this is due to gradient interference where the two modules may inadvertently oppose each other’s objectives, particularly in the early stages of training. In contrast, alternating training separates the updates: RAM is optimized solely with the concept alignment loss, and SAM is updated in a separate step using the semantic loss, conditioned on RAM’s output. This decoupling enables each module to specialize more effectively, reduces interference, and results in more stable optimization. We also explore a **related variant using a shared transformation in the supplementary material (Section 7.7, Table 15)**, where we again observe that alternating, modular updates lead to improved performance. We will include a direct comparison to joint training in the final version of the paper.\n\n### **Q2. Regularization to keep RAM small**\n\nRAM serves as the steering component that shifts the latent representation toward the target responsible concept, while SAM acts as a regularizer that preserves alignment with the original model’s trajectory. We do not impose an explicit constraint to keep RAM small because (i) the required magnitude of transformation varies by concept and (ii) enforcing a small RAM norm could lead to under-steering, reducing the effectiveness of fairness or safety interventions.\nInstead, we regularize the composite transformation via the semantic loss (weighted by $\\lambda$), which keeps the overall trajectory close to the original model.  \n\n **Shared transformation (Same module for RAM and SAM) :** We would like to clarify that the shared transformation design **has been evaluated in our supplementary material (Section 7.7; Table 15)**. Specifically, training a single transformation module using the combined loss underperforms compared to our proposed two-module setup (RAM + SAM) across all metrics. We believe this is because concept steering and semantic alignment are orthogonal objectives. When both are optimized through a single shared transformation, their competing gradients interfere, preventing effective specialization and resulting in suboptimal performance on both fairness and fidelity. In contrast, our modular design separates responsibilities: RAM applies strong, concept-specific transformations, while SAM maintains semantic consistency by remaining close to the identity. This decoupling enables each module to specialize effectively, resulting in improved performance across both dimensions. Additionally, such control is especially valuable for real-world deployment and interpretability, where the ability to tune each component independently is highly desirable. Achieving this level of modularity and transparency is challenging with a shared transformation approach.\n\nWe thank the reviewer once again for their valuable feedback and hope that our responses have addressed all the concerns raised."}, {"review_id": "lbucSqwYBP", "reviewer": "Reviewer_skB6", "summary": "This paper introduces RespoDiff, a new framework designed to improve the fairness and safety of text-to-image diffusion models without compromising image quality or semantic accuracy. The core of RespoDiff is a dual-module transformation applied to the intermediate bottleneck representations of a diffusion model. To train these modules, the authors introduce a novel score-matching objective that effectively coordinates their actions.", "questions": "- **Q1**: The paper demonstrates its method using fixed, general neutral prompts like \"a person\" for fairness and \"a scene\" for safety. However, in a real-world deployment, the model would receive diverse and complex user queries (e.g., \"a group of friends having a picnic\"). For such queries, a fixed neutral prompt like \"a person\" might lose significant contextual information. How does the RespoDiff framework envision handling arbitrary user queries?\n- **Q2**: For handling multiple concepts simultaneously (e.g., for intersectional fairness), the paper proposes a simple summation of the individual transformations. Have the authors investigated whether the effectiveness of one transformation change when combined with another?\n- **Q3**: The paper introduces a crucial hyper-parameter, $\\lambda$,  to weight the semantic loss. However, the paper does not discuss how this hyper-parameter was chosen. Could the authors provide an ablation study of the hyper-parameter?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- **Balance between fairness/safety and fidelity**: Unlike previous methods that struggle with image quality, RespoDiff introduces the Semantic Alignment Module (SAM) and a score-matching objective to preserve visual quality and semantic coherence.\n\n- **Thorough Evaluation**: The paper substantiates its claims with a thorough and rigorously designed evaluation.\n\n### Weaknesses\n\n- **Lack of Clarity**: While the paper describes RAM and SAM as transformation modules (e.g., main text in Section 4 and visual demonstrations in Figure 1), the actual implementation is an *input-agnostic* constant function that is linearly added. This is a bit overstatement and could cause confusion to normal readers.\n\n- **Lack of Novelty**: Based on the previous point, the transformation module is essentially parameterized as a constant function, which is too naive. Furthermore, this architecture is almost identical to the previous work *SDisc* [Li et al., 2024].\n\n- **Poor performance on different model**: While the method was applied to SDXL, the improvement in fairness was not as pronounced as with the v1.4 model, and it came at a higher cost to image fidelity. This suggests the method's effectiveness may not scale perfectly with newer, more complex model architectures.\n\n- **Limited applicability**: See Questions Q1 and Q2.", "was_revised": true, "final_justification": "My initial concerns centered on lack of clarity, novelty compared with the prior work SDisc, and practical applicability. However, the authors' rebuttal explanations and experiments effectively addressed these issues. Therefore, I decide to raise my score to 4.", "rebuttal": "We thank the reviewer for the valuable feedback and thoughtful comments. We address each of the concerns below:\n\n### **W1. Clarity**\nWe present RAM and SAM as general transformation modules to keep the framework flexible and extensible. While our main experiments use constant vectors for simplicity and strong empirical performance, **our training framework and losses are architecture-agnostic**. As shown in Section 7.6 of the supplementary, we ablate alternative architectures such as MLPs and convolutional layers, and find that constant transformations work best in our setting. In short, RespoDiff’s contributions do not rely on any specific parameterization. RAM and SAM can be replaced with richer, input-dependent modules if desired, and our ablations confirm the method remains effective under such changes. We will clarify this in the final version.  \n### **W2. Novelty**\nWe appreciate the reviewer’s concern. While it is true that our implementation uses constant transformation vectors similar to SDisc, our contributions are conceptually and technically distinct in several key aspects:  \n- **Score-Matching Objective:** SDisc optimizes a reconstruction loss by generating target concept images, adding noise, and training a vector to denoise using the neutral prompt. In contrast, RespoDiff introduces a score-matching objective that explicitly models trajectory shifts between the neutral and target prompts using denoised latents. This provides more direct and structurally grounded supervision for learning responsible transformations. We highlight this difference in Line 104 of the main paper.  \n- **Explicit semantic fidelity preservation:** A core novelty of our work is the Semantic Alignment Module, which explicitly regularizes the generated trajectory to remain close to the original diffusion model. This is an issue not addressed in SDisc, which lacks any such semantic fidelity constraint.\n- **Modular training:** We decouple concept steering (RAM) from semantic fidelity preservation (SAM) using dedicated score‑matching and semantic objectives with alternating optimization. This modular design is a key novelty of our approach and distinguishes it from SDisc’s single‑vector reconstruction setup. \n- **Empirical Performance:** RespoDiff consistently outperforms SDisc on fairness and safety metrics (Tables 1–2) while maintaining strong alignment. Qualitatively (Fig. 6; Sec. 7.4.5), SDisc often overfits to narrow stereotypes (e.g., sad or elderly faces, unrealistic profession depictions), likely due to its noise-reconstruction objective. In contrast, RespoDiff preserves profession context while effectively steering toward target concepts.\n\nWe believe that the above elements constitute a distinct and novel framework relative to SDisc. \n\n### **W3. Performance on SDXL**\nWe would like to clarify that while the fairness gain with SDXL may appear less pronounced in absolute terms, it is still a substantial relative reduction (Deviation Ratio drops by **~64%** from 0.72 to 0.26). This is a significant improvement considering that SDXL is a more complex and larger model. Additionally, the slight increase in FID (+0.95) is marginal and still within high-quality generation bounds, especially when weighed against the large fairness gains. Importantly, the alignment scores (CLIP and WinoAlign) remain virtually unchanged or even slightly improved. To further validate RespoDiff on SDXL, we now include safety experiments showing that it effectively removes inappropriate content while preserving strong image fidelity.  \n\n| Approach   | I2P (↓) | FID(30K) (↓) | CLIP(30K) (↑) |\n|------------|---------|------------|-------------|\n| SDXL       | 0.34    | 13.68      | 32.19       |\n| RespoDiff  | 0.17    | 13.90      | 32.10       |   \n\nWe also conduct additional experiments to investigate the effectiveness of our approach on a *non-UNet architecture such as Flux*. Due to resource and time constraints, we performed these experiments with Flux-mini. Since Flux uses transformer architectures without an explicit intermediate bottleneck space (as opposed to UNet architectures), we applied our modules directly to the text representations. Identifying a similarly interpretable latent space within transformers, as in UNet, required further exploration. We evaluated gender debiasing using this setup, and the results are summarized below. \n\n| Approach   | Dev Ratio (↓) | WinoAlign (↑) |\n|------------|----------------|---------------------|\n| Flux       | 0.71           | 24.73               |\n| RespoDiff  | 0.27           | 23.65               |\n\nThe experimental results demonstrate that our approach can effectively enable responsible generation even on architectures like Flux. We will also provide the qualitative results for Flux in the final version of our paper. We believe this highlights the versatility of RespoDiff beyond UNet-based models. Overall, our work includes extensive evaluations across diverse model architectures, going beyond prior responsible generation methods, to support the robustness and generality of our approach. \n\n### **Q1. Handling arbitrary user prompts** \n**Robustness to other neutral prompts:**  To assess the robustness of our approach to the choice of neutral prompt, and in response to the reviewer’s feedback, we conducted an additional experiment. We retrained the man and woman modules using an alternative neutral prompt “a group of people”, and evaluated them on two out-of-distribution prompts: “a group of friends on a picnic” and “a couple of doctors.” This allows us to test whether our method remains effective when trained on more complex neutral formulations. Due to the limitations of CLIP in capturing fairness and alignment for complex prompts, we employed GPT-4o for evaluation. We generated 200 images, uniformly sampling male and female modules, and used GPT-4o to:  \n- Classify each image as male or female (to compute a deviation ratio).  \n- Rate alignment to the prompt on a 1–5 scale. Images scoring ≥4 were considered well-aligned.\n\n| Prompt                         | Deviation Ratio (↓) | Alignment Accuracy (↑) |\n|-------------------------------|----------------------|-------------------------|\n| A group of friends in the picnic | 0.026               | 96.5%                  |\n| A couple of doctors             | 0.029               | 95.1%                  |\n\nOur results indicate that training with a different complex neutral prompt also yields balanced gender representations on arbitrary unseen queries such as ‘A group of friends in the picnic’,  while preserving prompt alignment, suggesting that the approach generalizes to other generic neutral prompts. We will present the qualitative results that support our analysis in the final version of our paper. \n\n**Handling arbitrary user queries:** As demonstrated in our experiments, our approach generalizes well across unseen contexts using a small set of general-purpose neutral prompts (e.g., “a person,” “a group of people”). To identify the neutral prompts, we envision training RespoDiff modules on a limited set of such prompts. At inference time, the system can select the most appropriate trained modules by computing similarity between the arbitrary user prompt and the available neutral prompts. For instance, a user prompt like “a group of friends on a picnic” would naturally align with a module trained on “a group of people.” \n\n### **Q2. Intersectional fairness**\nIn our intersectional fairness experiments (gender and race), we noticed some interference affecting gender more significantly than race. We believe this arises because we initially apply unit scales to both concepts. When composed, the concept with the larger effective residual can dominate, under‑steering the weaker one. To probe this, we varied per‑concept steering scales during composition (e.g., slightly >1 for gender and slightly <1 for race).\n\n**Evaluation of Gender fairness before and after Gender, Race composition**\n\n| Setting                          | Dev Ratio (↓) | WinoAlign (↑) |\n|----------------------------------|-----------|----------------|\n| Gender only                      | 0.14      | 27.30          |\n| Gender + Race (Scales 1, 1)      | 0.20      | 27.12          |\n| Gender + Race (Scales 1.1, 0.9)  | 0.15      | 27.09          |\n\n**Evaluation of Race fairness before and after Gender, Race composition**\n\n| Setting                          | Dev Ratio (↓) | WinoAlign (↑) |\n|----------------------------------|-----------|----------------|\n| Race only                        | 0.16      | 27.53          |\n| Gender + Race (Scales 1, 1)      | 0.14      | 27.12          |\n| Gender + Race (Scales 1.1, 0.9)  | 0.16      | 27.09          |\n\nAs shown in the tables, light concept‑specific scaling rebalances the composition, restoring gender steering to near its single‑concept behavior without significantly changing alignment, while race fairness remains stable. We will include these results and qualitative examples in the final version. Additionally, learning the scaling factors dynamically to compose concepts is an interesting direction to explore in future work.\n### **Q3. Sensitivity to $\\lambda$**  \nWe now present results on the sensitivity of RespoDiff to the hyperparameter $\\lambda$.\n\n| λ  | Dev Ratio (↓) |  WinoAlign (↑) | FID (30K) (↓) | CLIP (30K) (↑) |\n|------------|--------------|------------------|------------------|-------------------------|\n| 0          | 0.12         | 26.12            | 15.63            | 29.93                   |\n| 0.5 (Default) | 0.14      | 27.30            | 14.91            | 30.67                   |\n| 4          | 0.29         | 27.53            | 14.17            | 31.24                   |\n\nEmpirically, increasing $\\lambda$ keeps the trajectory closer to the base model, improving fidelity but weakening target‑concept steering, while decreasing $\\lambda$ has the opposite effect. We therefore use $\\lambda = 0.5 $ that balances fairness and fidelity."}]

Author rebuttal:

Meta-review: This paper proposes a framework for responsible and faithful text-to-image generation via dual-module bottleneck transformation in diffusion models. The proposed method addresses a critical challenge and outperforms existing baselines. All the reviewers lean to accept this paper. Therefore, I recommend accept. Please further revise the paper based on the reviewers' comments.
Decision: Accept (poster)