ProReGen: Progressive Residual Generation under Attribute Correlations
Authors: Ruby Shrestha
Ajay Gopi
Casey Meisenzahl
Bipin Lekhak
Linwei Wang
Venue: ICLR 2026 Poster (2026)
Keywords: attribute correlation
progressive training
data generation
Primary area: generative models

Abstract: Attribute correlations in the training data will compromise the ability of a deep generative model (DGM) to synthesize images with under-represented attribute combinations ($\textit{i.e.,}$ minority samples). Existing approaches mitigate this by data re-sampling to remove attribute correlations seen by the DGM, using a classifier to provide $\textit{pseudo-supervision}$ on generated counterfactual samples, or incorporating inductive bias to explicitly decompose the generation into independent sub-mechanisms. We present ProReGen, a $\textit{progressive residual generation}$ approach inspired by the classical Robinson's transformation, to partial out from an image attribute $\mathbf{x}_2$ its component $m(\mathbf{x}_1)$ that is predictable by other image attributes $\mathbf{x}_1$, and the residual $\gamma = \mathbf{x}_2 - m(\mathbf{x}_1)$ that is not. This simplifies the problem of learning a DGM $g(\mathbf{x}_1, \mathbf{x}_2)$ conditioned on correlated inputs, to learning $\tilde{g}(\mathbf{x}_1, \gamma)$ conditioned on orthogonal inputs. It further allows us to progressively learn $\tilde{g}$ by first shifting the burden to abundant majority samples to learn $\tilde{g}(\mathbf{x}_1, \gamma = 0)$, and then expanding it with additional layers $g\_{\text{res}}$ to resolve its difference to $\tilde{g}(\mathbf{x}_1, \gamma)$ using residual attribute $\gamma$ on limited minority samples. On three benchmark datasets with curated varying strengths of attribute correlation and one dataset with natural attribute correlation, we demonstrate that ProReGen---with input orthogonalization and progressive residual learning---improved the correctness of minority generations compared to existing strategies.

Published as a conference paper at ICLR 2026
PROREGEN: PROGRESSIVE RESIDUAL GENERATION UNDER ATTRIBUTE CORRELATIONS
Ruby Shrestha, Ajay Gopi, Casey Meisenzahl, Bipin Lekhak & Linwei Wang Rochester Institute of Technology, Rochester, NY, USA 660 riu.edu
ABSTRACT
Attribute correlations in the training data will compromise the ability of a deep generative model (DGM) to synthesize images with under-represented attribute combinations (i.e., minority samples). Existing approaches mitigate this by data re-sampling to remove attribute correlations seen by the DGM, using a classifier provide pseudo-supervin supervision on generated counterfactual samples, or incorpo- rating inductive bias to explicitly decompose the generation into independent sub- mechanisms. We present ProReGen, progressive residual generation approach inspired by the classical Robinson's 's transformation, to partial out from an image attribute X2 its component mx1)) that is predictable by other image attributes x1 and the residual 7 X2 mm(x)) that is not. This simplifies the problem of learning a DGM g(x1, X²) conditioned on correlated inputs, to learning @(x1, 7) conditioned orthogonal inputs. It further allows us progressively learn 9 by first shifting the burden to abundant majority samples to learn ( 0), and then expanding it with additional layers fres resolve its difference (x1; 7) us- ing residual attribute 7 on limited minority samples. On three benchmark datasets with curated varying strengths of attribute correlation and one dataset with natu- ral attribute correlation, we demonstrate that ProReGen-wiid -ithh input orthogonal- ization and progressive residual craninggimproved the correctness of minority generations compared to existing strategies.
INTRODUCTION
Attribute correlations are not uncommon in observed image datasets. Some may be a natural man- ifestation of underlying causal relations, e.g. the object in an image determining the background (Sagawa et al. 2019). Some may reflect bias in data curation, e.g., collecting patient data from those who already received treatment (Wang al. 2017). Regardless of the mechanisms, attribute correlations can induce unintended consequences in deep neural networks (DNN) training.
In the context of discriminative (e.g., classification) DNNs, this phenomenon is widely discussed, often under the concept of spurious correlations or short-cut learning (Ye et al.) 2024). In the con- text of deep generative models (DGM), such discussion comparatively less structured and scatters across a variety of topics. On one hand, DGMs are used in many domains to synthesize under- represented image examples-those with image attributes that do not comply with the observed correlation, e.g., for explaining whether DNN classifier has captured correlated features for deci- sion making (Rodriguez al., 200T]), or for augmenting training data to mitigate correlations (Kim) et al. (022). On the other hand, several evaluation studies (Träuble et al. 2021: Bose et al.) 2022) have shown that naively-trained DGMs would capture latent attribute correlations from training data (Träuble al. 2021; Bose al. 2022). How does this impact the synthesis of under-representedd samples, and to what extent could it be mitigated? Answers to these questions remain open.
Consider two sets of image attributes x and X2 (both can be multi-dimensional) that exhibit correlation in observed data. Consider the goal of learning a DGM conditioned on these attributes to generate an image y y = g(X1, X²). For the function g to generate with different combinations of x, and X₂ values, it is important for g to correctly model the mechanisms, 91 and G2. through which x, and x2 influences y separately. Unfortunately, due the observed X1-X2 correlation, 9

---

Published as a conference paper at ICLR 2026
x2 - mxi))
gree (Ranir (3) X1, ()
A.
B.
C.
D.
X1
X1
y
X
y
y
m(xi
2
Buir
X2
γ
g(x1, X2)
g(x1, 7)
Imj
Figure 1: A-C: Overview of the Robinson's 's partialling-out approach in motivating learning g(X1, x2) as 7) D: Conceptual illustration of ProReGen realize function g(x1, 7) by progressively learning 9mjr(x:) == 7 7 = 0) from majority samples, followed by learning Ires Ires(( (hnje (x) X1, 7) on minority samples to resolve the difference g(x1,7) with 7.
and 92 can only be separately observed in the small number of samples where such correlation does not hold. We stress this as a fundamental challenge for learning a DGM under attribute correlations.
Existing strategies to address this challenge are limited. Re-sampling is simple approach to balance training samples (Monteiro et al. 2022), essentially up-weighting under-represented samples where 91 and 92 can be separately observed. Alternatively, inductive bias has been introduced to explicitly decompose into independent mechanisms 91 and G2, which requires prior knowledge about how and X2 may influence y differently (e.g., object shape vs. texture vs. background) (Sauer & Geiger, 2020). Finally, to go beyond the limits of factual under-represented samples, gheudo-supervision on generated counterfactual images has been presented, typically realized by using a classifier to recognize feature attributes in the generated images (Kocaoglu et al., 2017 Ribeiro et al. 2023 He iet al. 2019). However, since the classifier is trained under the same attribute correlations, its ability to correctly recognize these attributes is likely compromised how does this impact its validity to supervise the generation of under-represented counterfactual images has not been well understood.
In this paper, we take fundamentally different perspective to address the challenge of modeling X2) under attribute correlations, inspired by the classical Robinson's 's partialling prriialiing-utt out trans- formation (Robinson 1988). While details of this concept will be introduced in Section 3 Fig. illustrates its core concept in the context of modeling g X1, X2). Consider the causal graph in Fig. with a causal direction assumed between X1 and xg. Instead of attempting to model the independent causal mechanisms 91 and g2 in the presence of such correlations, we decompose X2 into Exx x1]
that can be predicted by X1, along with residual y that cannot, i.e., x₂ E X2 Xx] + y (Fig. B)
With this, instead of modeling y =g(X,, X2) as composition of 91 and 92 as in Fig. IA we model it as y ?)) as Fig. [C: the effect of x, on y now absorbs the effect from Exx the component of X2 that can be predicted from XI; we referred to this as correlated effect; such, the effect of y on y-the part of X2 's influence on y that cannot be explained by xI partialled out: refer to this as residual effect. In this new causal graph (Fig. IP), instead of attempting to recover two independent mechanisms from correlated inputs X1 and X2, we transform the problem into learning independent correlated and residual effects from two independent inputs x; and 7-
To design a DGM based on Fig. DC. we note that [X1, 7) at 7 = 0, i.e., Ely|x1, 7 = 0), can be estimated from abundant samples whose attribute values meet the observed correlations (henceforth referred to as majority samples). g(x1, 7) at + O. on the other hand, can only be estimated from limited number of samples where such correlation does not hold (henceforth referred to minor- ity samples). To shift the primary burden of learning ğ to majority samples, therefore, we further decompose the learning of g(x1.?) into the learning of g(x1, 0) at ? 0 using majority samples, and use minority samples to make up the difference between g(x,, 0) to 7 This results in our progressive residual effect generator (ProReGen) that is progressively learned in two stages as outlined in Fig. [D.. ProReCien as described is expected to have two major benefits. First, the or- thogonalization of the inputs x and 7 helps separate the learning of their independent effects on y. Second, with the progressive expansion from Jmir to gress the challenge of learning to generate under attribute correlations is reduced to learning the residual between ğ (x11.7 y = 0) and ğ(x}, 7). We instantiate the concept of ProReGan in conditional- -aaiitiionalVAAs,,, -GANs, and -diffusion models (DMs). On three benchmark datasets with varying strengths of attribute correlations and one dataset with natural attribute correlations, we experimentally demonstrate the improved performance of ProRe-
2

---

Published as a conference paper at ICLR 2026
Gen in generating correct minority samples compared to naive DGMs or those strengthened with re-weighting of factual samples or sueudo--upervision of generated samples.
2 RELATED WORKS
Several domains find uses for synthesizing minority images. For instance, to explain if classifier has captured attribute correlations from its training data, one can test if it is able to guide a pre-trained DGM to generate minority images (Rodriguez
al., 2021: Jeanneret al. 2022). In addition to
explaining, synthesized minority images can also be used for augmenting and removing attribute correlations in training data (Goel et al. 2020 Kim et 202T). While saively-trained DGMs have been used for such syntheses (Rodriguez et al., 22021 Jeanneret et al., 2022), there is an increasing attention on the impact of attribute correlations on DGM training and potential mitigation solutions.
we-weighting of factual samples: The concept of simulated intervention was presented in Monteiro al. (2022) to re-sample data according to the marginal distribution of image attributes, to effec- tively remove attribute correlations seen by the DGM. This is essentially similar to re-weighting, where minority samples are up- up-weighted in their contribution to the training signals. This approach is ultimately affected by the quantity and diversity of factual minority samples.
Pseudo-labeling of counterfactual generations: Instead of relying on factual minority samples, an alternative is to provide some seeudo-ttpervision to encourage the DGM to generate counterfactual images with intended feature attributes. This is often achieved by leveraging another DNN classifier, often trained from the same data as the DGM, to provide supervisory signals by recognizing the attributes of generated images (Kocaoglu et al, 2017] Ribeiro et al. 2023 He et al. 2019). However, because the DNN classifier is also subject to attribute correlations in the training data, their reliability in correctly recognizing these attributes is questionable. How does this affect the supervisory signal it provides to the DGM's 's counterfactual generations has not been systematically investigated.
Inductive bias to decompose generation mechanisms: An entirely different approach is to incor- porate inductive bias about the mechanisms under which different attributes contribute to generated images (Sauer & Geiger 2020; Park et al. (020)) In (Sauer & Geiger 2020), for instance, the image generation process is decomposed into independent shape, texture, and background mechanisms. While this approach tends to be highly successful when assumptions of the underlying generation mechanisms are met, it does require prior knowledge for the design of such inductive bias.
Disentanglement under attribute correlations: More broadly, several evaluation studies have shown that naively-trained DGMs would capture attribute correlations entangled representations Träuble et al. 2021) and even inherit the associated causal directions (Bose et al., 2022). To learn disentangled represenations from correlated attributes, existing works in VAEs e.g., encourage pair- wise factorized support by minimizing hausdorf distance (Roth et al. 2023). minimize conditional mutual information between subspaces with respect to categorical variables (Funke et al.) 2021), or extend contrllllle VAEs to disentangle groups of properties while allowing correlations within the group. Similarly, in GANs, works exist to navigate GAN's latent space to disentangle otherwise entangled attributes, e.g., by latent optimization (Li al. 2020) or leveraging gradien information (Chen et al., 2022). In general, these works focus on disentangled representations or image manipulation, rather than generation of minority samples as ProReGen.
Generative modeling under attribute imbalance: Finally, ProReGen is related to generative mod- eling under attribute imbalance (Georgopoulos al. 2020 Zhao al.] 2018 Haider et al.) (025)
although the challenges they tackle differ: while imbalance presents a general generalization chal- lenge for the model to represent rare attributes, correlated attributes require methods to explicitly handle dependence among co- co-occurring attributes as specific form of distribution shift.
ProReGien represents a completely different approach to these existing works. Inspired by the clas- sical Robinson's 's partialling out approach, ProReGen tackles the challenge of attribute correlations at its core by first recasting the DGM from being conditioned on correlated attributes to orthogo- nal attributes. By a progressive expansion design, it further leverages majority training samples to reduce the problem of learning minority generation to learning its residual to majority generation. The latter concept is related image translation models that transform factual majority sample to minority counterfactual (Kim et al. 2021 Goel et al. 2020), An et al. (202)). Because these models intended only for translating factual samples, they out of the scope of our consideration.
3

---

Published as a conference paper at ICLR 2026
3 PRELIMINARY: ROBINSON'S PARTIALLING-OUT TRANSFORMATION
Consider a partial linear equation Elylar, x2] = 0(x1) + Brg, where z2 = m((1) + uu, Robinson's transformation in [Robinson| 1988) decomposes the original equation into:
B B 11, 22] = ((21) + 3 (m(r1) + u) = E' [1] + 3 (I2 m(x1))
(I)
where Ely m1| = ((11) ++mm()++))))))) Effectttiiee instead of describing the separate effect of æ1 and Effectively,
22 on y as dictated the original equation, their effect is decomposed into two orthogonal components:
1) Elyl that describes the combined effect of I1 and m(x1) on y y-the latter absorbing the effect
of 22 on y that can be predicted by 21; and 2) the residual effect of / (x2 - m(x1)) from x2 m(x1))thhe effect on y from the residual of 22 that cannot be predicted from 2
In the original paper (Robinson 1988). the goal of this decomposition is to estimate parameter 3. which arrives at the classical rsiiuull-o--rsiddue
least-square fitting of 3 via: Eiy z2l]
Ely|z1] 8 (I2 - E[zg|z1]), The resulting estimator for 3 can further be shown to meet the Neyman's 's orthogonality condition such that it is insensitive to perturbations in the estimator for Fly]ar]] or F[ [xi] (Robinson 1988, Chernozhukov et al., 2018). This decomposition has also served as the basis for the R-leamer (R for Robinson) that extended the approach to estimating the function ((11) instead of the low-dimensional parameter 3 (Nie & Wager 2021
In contrast, we use this transformation the foundation inspiring the design of an image-generating DGM, where y is high-dimensional and its relation with æ1 and 22 is highly nonlinear. Note that, in the original formulation, the causal direction ø1 - I2 depends on the target parameter 3; in our setting where y as a function of g remains the primary interest, the assumed causal direction will influence the decomposition, but not the general modeling strategy. The difficulty of the residual generation task however may change with this direction, which we will empirically examine in Section 5.3 Below we continue discussion with 2 - I2 without loss of generality.
4
METHODOLOGY
4.1 PROREGEN: PROGRESSIVE RESIDUAL GENERATION
Consider conditional-DGM y = g(z, X1, x2) underlying observed image data, where a correlation exists between image attributes x and X2, and z represents latent variables not included in and X2. Built on Robinson's is partialling-out approach, as illustrated Fig. [ we first cast the problem of learning y g(z, X2) with correlated inputs, to learning y ğ(z, x1,7) with independent inputs where 7 = X2 m(x1) represents the residual in X2 that cannot be predicted by X1. Because x1 = 0) is described by majority samples vs. İ(z, x1, 7) 7 of 0 by small number of minority samples, we further design a progressive learning strategy to shift the burden of learning Ò mostly to the learning of (Z, x1 7 = 0) using majority samples, and using minority samples to only resolve its difference to gZ, X1. 7) with 7. This gives the foundation of ProReGen:
y g(z, x1, 7) Re gres (bmir (x1 X1, x1,7), where ? y= X2 - m(x1)
(2)
where hmje the feature map of Imjr = g(z. X1, y 0) before the final activation layer. Equation
2) includes three main components progressively learned in two stages, as illustrated in Fig. DD:
In stage-I, from a large number of majority samples, we learn an attribute predict function X2
m(x1) to approximate F[xa|x1], [x]]. and generative model Mmjr (x: to approximate ğ(x1, = 0), the latter effectively describing the generation when X2 = mx()), iee., for a majority sample.
In stage-II, using available minority samples, we expand the generator with additional layers, Gres. to resolve the residual between ğ g(x,,7) 7)) and (mj-(X)) with the residual y partialled out from X2- Effectively, approximate ((),,7) by gres (hmjj (x1),x1,7) with hmjr defined above.
This takes form of a progressive-DGM, where a backbone Gmjr is first learned on majority samples and then expanded on minority samples. This concept is agnostic the type of DGMs: below, we describe its instantiations on conditional-VARe (c-VAEs), -GANs (c-GANs), and -DMs (c-DMs).
4.2 PROREGEN- VAE
Stage I: To learn Imje (z, X1) as a VAE, we define a decoder network Gom. (z. X1) that parameter- izes the likelihood po... (ylz, x1), and its corresponding encoder network Feg., (1) that parame-
4

---

Published as a conference paper at ICLR 2026
terizes the approximate posterior Iam (zly, X1), both conditioned attribute labels X1. They are trained on majority samples by maximizing the standard ELBO loss. A function X2 = xx)) is estimated on attribute values from majority samples.
To leverage the learned 9mir (z, x1), we now expand the decoder Gamy with several addi- tional layers before the final activation layer o, which we denote as Go We denote this expanded portion of the decoder as Gt.,, (Go \o, x1: conditioned on 7 = X2 ri((x1). As illustrated in Fig. [AA, expand the encoder network with additional layers E in mirror of the expanded decoder network as Esg. (Ymar: 7), to produce output that will serve as the input to the first-stage
Stage I:
encoder. The expanded networks are trained minority samples by maximizing the ELBO loss below, where we keep the stage-I weights mnj and Onje frozen:
max FBx=Fem (Fror ())))))))))) ||lmmm Ymar [|21
+ B DKL(Egu (E (yo (ymmr y), x1, p(z))
(3)
KL. Divergence
where Ymnr G (G (z, x1), X1, ?),
2 Exs (Err (Y (ymr x1)
(4)
While Equation (4) represents a general formulation for residual generation, on simpler datasets, an additive residual can be considered such that ymnr Go x1) Go (Gomg) Va (2.x,)), 7)- With Equation 44 Ga., leverages the feature map of the stage-l generator Go... and the residual to resolve the difference between majority and minority generations. Intuitively, because Go... and E are trained in stage-I to generate and encode from majority samples, the expanded Go... will be encouraged to learn modify a majority-image feature map to include features corresponding the residual partialled-out from X2, while the expanded Eg... will be encouraged to alter such features to generate an output feature map acceptable Eg.,, (i.e., compliant with feature map seen by E in stage-I). With this, we shift the burden of learning the c- VAE mainly to majority samples, and allow the use of limited minority samples for learning the necessary reisdual changes only.
4.3 PROREGEN-GAA -GAN
Stage I: To learn Gmjr(z, x1) as c-GAN, we define generator Geay (2,x1) and discriminator D dry (y.x,) both conditioned on attribute x,. They are trained on majority samples using standard adversarial loss. A function X2 mxxx) is estimated on attribute values from majority samples.
x1)
Stage II: Similar to the setting of c-VA,, we now expand the generator Ga., with several additional layers, denoted as Go... starting with the feature map produced by Go... before the final activation layer. As illustrated in Fig. 6B. we expand the discriminator with additional layers Den.. in a mirrored fashion. Both Ge.,, and De are conditioned on y = X2 -m((x)) The expanded networks are trained on minority samples using the adversarial loss while freezing stage-l network weights Omjr and Omir:
min max {E Xi, log D Do (Dom. (ymne, 7), x1)]
Pex-p())xxx [log (1 D (D ($ mnr; 7), x })]}}
(5)
(6) Similarly, with Equation (6). the expanded Ge.. learns use the residual attribute y to change the distribution of generated majority samples to one that aligns with the distribution of real minor- ity images Ymm- At the same time, the expanded discriminator Den.. is encouraged to change the residual features on the real/generated minority sample in order to leverage the stage-I discriminator Dony that has learned to work with the distribution of majority samples.
where Ymnr = Go (G x x1, 7),
10,
One difference between Equation for ProReGen-Van VAE and Equation (6) for ProReGen-GAN is the distribution over which the sample is taken. This is inherently determined by the training loss of the two models, where the likelihood loss of VAE is calculated over the posterior distribution of z conditioned on an observed image (emphasizing instance-level reconstruction) vs. the adversarial loss in GAN is calculated over the prior density of z (emphasizing distribution-level distance).
4.4 PROREGEN-DM
Stage I: We follow the standard denoising diffusion probabilistic model (DDPM) Hoo et al. 2020 to learn c-DM that capture attribute correlation. We noise the majority samples Ymjr, X
5

---

Published as a conference paper at ICLR 2026
using the closed form noising expression in Equation derived from the forward noising pro-
cess q(Ymje, Xmir.-)) := (Vymjr, -I-
Vmjt, Sejee,
where {Br is the noise schedule.
To obtain Gmir (z, X1), we learn the reverse denoising process parameterized by EBmy (Ymjr, f, x1), conditioned on attribute label X1, using the loss function in Equation (8).
Y: = Vo + v1- 5,
e~ (0,
Gt II Cs. at = 1 Bt
(7)
LMSE (f) = Eyo.t.e [1le co co 1, (
where
X1 (Stage-I) o (Stage-II)
(8)
Given the unique design of DM, we cannot simply expand the learned denoising process in stage-I with additional denoising steps. Instead, as illustrated in Fig. Fig. we introduce a trainable stage-Il network 6B... (y m'nr 1.7 7) conditioned on the residual attribute 7, trained with only minority samples using Equations 7 but injected with features learned by the stage-I cΦ... (Ymjr, x1) this is inspired by the design of control-net (Zhang et al., 2023), but distinct in that we the fronzen stage-I network to provide features for the stage-Il network. We consider Ymjr, input of coap as a corresponding majority version of Ymmr, with all aspects except the residual feature identical. We inject stage-I features the downsampling and middle blocks of U-Net (Ronneberger et al.) 2015), but not in the upsampling blocks as we empirically observed better quality without it. We use the same variance schedule, diffusion steps, and architecture design in stage-Il network so that the features coming from stage-I are aligned to the particular diffusion step of stage-Il being processed.
Stage II:
EXPERIMENTS AND RESULTS
Data: For quantitative evaluations that require oracle classifier trained balanced attributes, we consider
(Lee et al., 202I), MNIST-Correlation (Mu & Gilmer (019). and Comupted CIFARRIO (Hendrycks & Dietterich] 2019). We assume known labels of the attributes that are correlated. For the two MNIST datasets, we curated high levels of correlation strengths 95%, 98%, 99%, and 99.5%, where the % represents the percentage of majority training sam- ples in the training data. For Corrupted-CIFARIO derived from natural images, we considered a wider range of correlation strengths of 70%, 80%, 90%, and 99%. For each dataset, we included a balanced version without any attribute correlations to both establish oracle attribute classifiers and establish a reference performance for all models considered. To test on datasets with natural attribute correlations, we considered CelebA for qualitative evaluation (due to the lack of an oracle classifier). Details of the dataset are described in their respective sections below. Test accuracies of their oracle classifiers are in Appendix B and their training data distribution in Appendix C
We considered e- VAE, c-GAN, and c-DM baselines with the following strategies for mitigating attribute correlations: 1) naive, 2) re- re-weighting, achieved with upsampling minority sam- ples using the weighted random sampler and 3) pseudo-supervision on counterfactual generations, represented by causal-cHVAE (Ribeiro et al. 2023) where classifier is used to finetune the model in an optional second-stage stage of counterfactual generation, and causal-GAN (Kocaoglu et al., (017)) where the attribute classifier is incorporated in the end-to-end adversarial training. We could not identify causal formulations of DMs that accommodate multiple correlated attributes. We present the architectural details and parameter counts of all baselines and ProReGen in Appendix(D)
Baselines:
Evaluation: We evaluated the performance of all DGMs in generating both majority and minority samples. To generate with the trained DGMs, we sampled from z ~ (O, I) and generated a total 25,000 samples, with equal number of samples for each unique attribute combination, per dataset. We evaluated generated samples using: 1) correctness, measuring the ratio of generations in which the attributes evaluated by the oracle classifier match the intended attributes; 2) Fréchet Inception Distance (FID) (Heusel et al.] (017).. measuring the quality and diversity of generations by compar- ing the representations (retrieved from InceptionV3 network) of the generated samples against test set of diverse real samples; and 3) coverage & density (Nacem et al. 2020), J measuring the diversity and fidelity, respectively, of generations compared with a test set of diverse real samples.
5.1 EXPERIMENTS & RESULTS ON COLORED-MMIST
Settings of Attribute Correlations: Colored-CMNIL (Le et al. 202T is a commonly-use benchmark for synthesizing attribute correlations in the training data. It is MNIST-variant with

---

Published as a conference paper at ICLR 2026
Reesightng
Paauddoeupprttiion (cHusk-cHWA))
Profieden (Ours
Comrecnnss
Coverage
Majortt
Meoortyy
Majorty
Mhnorty
0375
56-
34-
0333
1.8
Conditona-ann
039
24-
0.4
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.7-
0.4
0.0
Bal
95%
98%
95%
95.5%
Bal
55%
96%
95%
99 516
Strnngth
B.
Nalve
Re-weighting
Naive
Re-weighting
ProReGen-GAN
0
0
4 3
c-GAN
f
8
0
8
4
4 8 8
4 6
3 2
G-VAE
Peaudo-supervision
ProRaGen-VAE
Neive
Rewniighting
ProReGen-DM
0
c-D1
$ 6
6
O O
0 4
0
4
8 8
7
9
5
5
4 4 8
g
8
8
Figure 2: A: Correctness and Coverage of majority and minority generations from ProReGen vs. baselines for ColOTed-MNIST. B: Visual examples of minority generations 99.5% correlation.
distinct majority color for each of the 10 digits. This creates a correlation between digit and color attributes, both discrete but non-binary. All basclines as described were considered in this dataset, with the exception of causal-GAN which was designed to work with binary labels (Kocaoglu et al.) 2017). We considered digit - color the causal direction 21 - 22 for experiments.
We present some representative quantitative metrics and visual examples on Colored-MNIST in Fig. Complete results included in Appendix K. As shown, ProReGen (green) in general improved the correctness of minority generations in comparison to the naively trained baseline (blue, significantly more in c-GAN and c-DM), at comparable quality metries. In comparison, while causal-cHVAE (red, the baseline that leverages pscudo-supervision) also im- proved the correctness of minority generations, this improvement was obtained at the expense of degraded correctness in majority generations, suggesting that the use of pseudo- superviiie has introduced trade-off in the correctness of majority vs. minority generations strengths. ProReGen, in comparison, was relatively stable across correlation strengths for majority generations. Note that the relatively strong quality metrics of causal-cHAVE may be due to its base hierarchical VAE architecture that different from the rest of the VAE models.
Results and Analysis:
higher correlation
Re-weighting resulted in comparable correctness metrics in both majority and minority generations in comparison to ProReGen. However, in all models, signs of overfitting could be observed in minority generations hence limited diversity, as shown in the rapidly-degrading Coverage metrics the correlation strength increased in Fig. 2 and the lack of diversity in the visual examples in Fig. B In re-weighted c- VAE, the higher coverage of minority generations in comparison to ProRe- Gen can seem counterintuitive. A potential reason that generations from learned distribution in ProReGen were comparatively more blurred compared to memorization of training samples.
5.2 EXPERIMENTS & RESULTS ON MNIST-CORRELATTION
Settings of Attribute Correlations: MNIST-Correlation Muu & Gilmer 2019) is another MNIST variant where most of the even digits are clean and most of the odd digits include zigzag, hence resulting in correlation between attributes x1 feven, odd} and x2 {clean. zigzag). We sim- ilarly created correlation strengths at 95%, 98%, 99%, and 99.5% following (Goel et al. 2020). Along with the information on presence / absence of zigzag, we also added the coordinates of
7

---

Published as a conference paper at ICLR 2026
Navv
Reawsighting
Pasusb-upparisaon (cauaalciivvee CHUBSi-GAR)
ProfteGan (Duni)
Correctnese
Majority
Minontt
Majority
Iirentty
1.00
56
Conditrrhya55
3:50
0.9-
2.50
0.2
0.99
20
831
9396
98%
95%
99.5%
Bal
55%
98%
99% 99.5%
CondioonalGaa
6.4-
100
0.5
0.7
0.4
2.6
02
Bal
95%
96%
90%
99.5%
Bal
05%
96%
90%
90.5%
Bal
95%
96%
90%
99.5%
Correaaiin Strongth
B.
4
3
8.
6 4
4 3
4
4 3
6
&
A L 0
D
8 2 7 6
8
1
/
7
6
9
/
9
3
7
7
7
7
1 2
5 6
9 4
8
P 4
3 θ
8
7 6
8 9
G
&
Y
O 4
6 1
4 3
8
9.
8
2
4
6
7
7
2 3
7
4
3
0
4.
9
9 9
Re-weighting
Peeud--auperiiion
Figure 3: A: Correctness and FID of majority and minority generations from ProReGen vs. baselines B: Visual comparison of minority generations at 99.5% correlation ratio. In each image grid, the intended generation is zigzag-even on the top, and clean-odd for the bottom.
for
the end points of zigzag (mid-point of image in case of clean image) additional feature tributes in X2 to represent residual attributes that cannot be predicted from x₁ but will contribute the generation of images. Additional details on this included in Appendix [ We consider even odd + presence / absence of zigzag as the causal direction for our experiments.
We present quantitative results and visual examples for MNIST-CCorclltion in Fig. 3 with complete results in Appendix K.2 c-DMs were not included because the highest level of correlation strengths did not induce bias in naive version. Compared naive VAE (blue), ProReGen- VAE (green) improved the correctness of minority generations, at some degradation in the correctness of majority generations and similar or slightly worsened quality metrics. compar- ison, causal-cHVAE (red) was inconsistent in improving the correctness of minority generation at more significance deterioration of both majority correctness and FID metrics. Re-weighting (purlec delivered similar correctness in minority generations and quality metries in majority generations, but better majority correctness and worsened FID (reflecting diversity issue) in minority generations.
Results and Analysis:
Compared to naive c-GAN (blue), ProReGen-GAN (green) significantly improved the correctness of minority generations along with significantly improved FID in both generations with moderate degradation of correctness in majority generations. Causal-GAN (red) was not successful in im- proving the correctness of minority generations, with FID metrics similar to the naive baseline. le-weighting (purple) improved correctness of minority generations with slight compromise in the correctness of majority generations, but also worsened FID metrics.
5.3 EXPERIMENTS & RESULTS ON CORRUPTED-CIFAL
Settings of Attribute Correlations: We adopted CIFARIO (Krizhesky et al. 2009 and cu- rated it following the practice in (Hendrycks & Dietterich] 2019) to create correlation be- tween object classes and image corruption types. More specifically, we considered five differ- ent object classes,
{car, bird, dog, horse, ship), and applied unique type of corruption, X2 = {gaussian noise, shot noise, impulse noise, contrast, brightness). respectively, to the majority of the training samples per object class. The minority samples have remaining corruptions uniformly sampled at random. We considered correlation strengths at the level of 70% and 80% for GAN-based models and 90% and 99% for DM-based models because they the levels which bias in-
8

---

Published as a conference paper at ICLR 2026
Naire
Re eeigiitingg
ProfeGee (Ours]
A.
Comeetneess
Caverage
Majoriy
Minorlyy
Majority
2.00
Minority
0.7
0.55
2.5
0.65
0.85
3.4
0.80
0ed
70%
B17
0ml
70%
03%
841
70%
80%
70%
80%
Canditoonkkou
0.5
0.8
0.4
0.6-
0.6
D4ll
90%
99%
Dal
90%
99%
Dal
90%
59%
96%
59%
Commtation Strongth
B.
Naive
Re-weighting
ProReGen-GAN
Naive
Re-nnighting
ProReGen-OM
Figure 4: A: Correctness and Coverage of generations on Corrupeed-IFFARO, B: Visual examples of minority generations 80% correlation for c-GAN and 99% correlation for c-DM. In each image grid, the intended generation is: top {car, impulse noise): bottom [horse, brightness}.
Naive
Re-weighting
ProReGen-DM
Malo
Female
Figure 5: Visual examples of minority generations on CelebA under natural correlations.
duced in the respective naive versions of these models. We did not test VAE-based models due to their quality issues in generation (blurring of corruption details) on these natural images.
Representative results in Fig. showed that both naive c-GAN and c- DM (blue) suffered from increasing correlation strengths, especially in the correctness of minority generation. Re-weighting (purple) sometimes addressed this issue (in re-weighted c-DM), but at the expense of decreasing diversity (c-DM) and deteriorated majority correctness (c-DM and c- GAN). In comparison, ProReGen (green) improved the correctness of minority generations, without compromising majority generations and with limited degradation image qualities.
Results and Analysis:
5.4 EXPERIMENTS & RESULTS ON CELEBA
Settings of Attribute Correlations: We considered multi-attribute face dataset CelebA (Liu al. 2015) and the natural correlation between X1 Gender and X2 = Hair Color; most blond-haired indi- viduals are Females (22.9K) with only around 1.3K blond-haired males in the entire training dataset. We considered four subgroups (Male/Female, Blond/Black Hair) with the naturally existing correla- tion the dataset. Additionally, we experimented with reducing the count of (Female-Bis Black Hair) subgroup to be close to that of "Male-Blond Hair). We tested only DM-based models here given their stronger performance in previous experiments.
Fig. 5 provide examples of minority generations for models trained under natural correlation, with additional results provided in Appendix K.4 Visually, naive c-DMs suf- fered evidently in the generation of minority images both in attribute correctness and generation quality. ProReGen- tribute combinations. While it is difficult to pinpoint its performance over re- w-weghtting without further quantitative analysis (due to the lack good oracle classifier), these results demonstrated the utility of ProReGen in realistic image datasets with natural attribute correlations.
Results and Analysis:
was able to consistently generate realistic minority images with correct at-

---

Published as a conference paper at ICLR 2026
Table 1: A comparison of progressive two-stage training vs. simultaneous training of Jmje and Pres in ProReGen-GAN for 95% correlation strength in Colored-MNIST
Correctness
FID
Coverage
Density
Two-Stuged Training
Majority
0,9592 + 0.004
16,9488
2.3446
0:9003 + 0.0275
0.7628 + 0.0386
Minority
0.9256 0,0257
17.2562
7,8816
0.7519
0.0903
0.6089 0.1216
Single-Stuged Training
Majority
0.929 0.0369
29.0843 + 6,3397
0.7636 + 0.0216
0.5585 + 0.1144
Minority
0.3557 0,1689
65.0227
12.3787
0.0432
0.0335
0.0320 : 0.0220
5.5 ADDITIONAL ABLATION STUDIES
To demonstrate the benefit of the progressive two-stage training, we performed an ablation of ProReGeN where the model architecture remained the same but Jmjr and fres were optimized simultaneous vs. progressively in two stages. As shown in Table without progressive training, there was minimal to no effect on the correctness of majority generations but some impact on its quality (e.g,, a drop of 27% in density). The generation of minority samples however was significantly worsened (e.g., drop of 62% in correctness and nearly three times worse in FID). We further demonstrate this with sample minority generations in Fig. in Appendix.
Effects of Progressive training:
Sensitivity to errors in mxx)): To understand the sensitivity of ProReGen to errors in the estimation of mxxi). we perturbed X1 to 8 via uniform random shift, within the valid domain, to simulate a wrong estimation of mxx)). )- We experimented with three levels of perturbation with increasing percentage of samples induced with random shifting attribute X1 per training epoch. We present our observation in Table 12 in Appendix G for ProReGen-GAN trained Colored-MNISS with 95% correlation strength. The correctness of ProReGen-GAll
dropped as expected with the increase in the level of perturbation, although not rapidly and still improved over the naive model at 80% of errors. A closer inspection showed that the drop resulted from inaccuracy in color generation, which as expected since 7 dictates the color residual between majority and minority samples.
Computation cost and additional training details: We present the computation cost, convergence plots, and small analysis of sensitivity residual sub-networks in Appendix H
Effect of assumed causal directions between attributes: We examined the effect of the assumption of causal directions between attributes by inverting the causal direction digit - color to color + digit for ColOTed-MNIST. We considered ProReGen-GAN for our analysis. We observed that the performance with the inverted causal direction color - digit was suboptimal, with only 0.0811 + 0.0127 correctness, on average, of minority generations vs. 0.9396 + 0.0016, on average, with digit + color. The correctness of majority generations was similar. This indicated that learning the residual for digit conversion was much more difficult. We present the generation samples along with additional results in Appendix [ This suggests that the difficulty, and hence performance of residual generation task, is influenced by the causal direction assumed and should be used to design the attribute causal direction for ProReGen in practice (unless the true causal direction is known).
CONCLUSIONS & DISCUSSIONS
We present ProReGen, a novel DGM-design that employs progressive training and leverages ma- jority training samples to learn most part of the generation task while employing minority training samples to only learn the residual information. We demonstrate its benefit in improving generation correctness against the baselines using synthetic and natural images different correlation ratios. Limitation: ProReGen-Vae VAE and -GAN perform residual operation at the image level. While ProReGen-DM leveraged feature injection, the base DDPM considered performs diffusion at the pixel space. Future works will investigate extending the concept of ProReGen to realize the effect of residual 7 on image generation through the latent space. ProReGien as presented assumes the ability to separate samples into discrete subgroups: future work will extend it to settings with continuous attributes and use attribute residual to modulate sample importance in stage vs. II learning instead. Evaluation of minority generation remains significant challenge: since it is not possible to have a perfect oracle classifier due to reasons such as natural attribute imbalance, future works should assess the uncertainty of these correctness metrics. Finally, the challenge of attribute correlation can be expected to persist and even amplify in text-to-image models due to the sparsity and heavy-tailed nature of the problem space, leaving exciting avenue for future explorations.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We present the design of network architectures and training details used in our proposed method in Appendix [D] Moreover, we provide references to the official code repositories employed for experimentation with two of our baselines in the same section. The source code for ProReGen is available at https::// / //gi thub com/ ruby stha/ wwchaaProeeeenneoee ProReGen_CCL220 share the training data distribution of the datasets used present our results in Appendix
Additionally, we
ACKNOWLEDGEMENTS
This study is supported by National Institutes of Health National Heart, Lung, and Blood Institute (NIH/HHBII) Grant ROIHLI45590.
REFERENCES
Jaeju An, Taejune Kim, Donggeun Ko, Sangyup Lee, and Simon S Woo. A 2: Adaptive Aug- mentation for Effectively Mitigating Dataset Bias. In Proceedings of the Asian Conference Computer Vision, pp. 4077 4092, 2022.
Joey Bose, Ricardo Pio Monti, and Aditya Grover. Controllable generative modeling via causal reasoning. Transactions on Machine Learning Research, 2022.
Zikun Chen, Ruowei Jiang, Brendan Duke, Han Zhao, and Parham Aarabi. Exploring gradient-based multi-directional controls in gans. In Proceedings of the 17th European Conference on Computer Vision (ECCV), 2022.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/dehiased machine learning for treatment and structural pa- rameters, 2018.
Christina M. Funke, Paul Vicol, Kuan-Chiel Wang, Matthias Kümmerer, Richard Zemel, and Matthias Bethge. Disentanglement and generalization under correlation shifts. arXiv preprint 1xXiv:212.114744 2021.
14754
Markos Georgopoulos, Grigorios G. Chrysos, Maja Pantic, and Yannis Panagakis. Multilin- ear latent conditioning for generating unseen attribute combinations. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119, pp. 3442-3451.
PMLR. 2020. URL georgopoul wwwp:ooppoooo2.......
Karan Goel, Albert Gu, Yixuan Li, and Christopher Re. Model patching: Closing the subgroup per- formance gap with data augmentation. In International Conference on Learning Representations, 2020.
Fasih Haider, Edward Moroshko, Yuyang Xue, and Sotirios A. Tsaftaris. Do generative mod- cls learn rare generative factors? Frontiers in Artificial Intelligence, 8: 8:1697139, 2025. doi: 203.8///raii.225519977771 artiFicial-Intelllleen http://aa..aaaaaaaaaaaaaa/////a. 2025. 101.
URL https://www http::/www.o.eeeeeeeoeorooreeeea 33.9//00..33//////
Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen. Attgan: Facial attribute editing by only changing what you want. IEEE transactions on image processing, 28(1 1):5464- 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv:1903. 12261, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.

---

Published as a conference paper at ICLR 2026
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic mod- els. In Advances in Neural Information Processing Systems (NeurIPS) 33, pp. 6840- 6851, 2020. URL https: #c5bcfecc5844 19019991111a1111111----be-00
pdf
Guillaume Jeanneret, Loie Simon, and Frédéric Jurie. Diffusion models for counterfactual explana- tions. In Proceedings of the Asian Conference on Computer Vision, pp. 858-876, 2022.
Eungyeup Kim, Jihyeon Lec, and Jaegul Choo. Biaswap: Removing dataset bias with bias-tail bias-tailored swapping augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14992-15001, 2021.
Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath. Causal- gan: Learning causal implicit generative models with adversarial training. arXiv preprint arXiv:1709.02023, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. Advances in Neural Information Processing Systems, 34:25123-25133. 2021.
Ziqiang Li, Rentuo Tao, Hongjing Niu, Mingdao Yue, and Bin Li. Interpreting the latent space of gans via correlation analysis for controllable concept manipulation. In Proceedings of the 25th International Conference on Pattern Recognition (ICPR) 2020, 2020. URL https::/
openreview wew.eoormridddteeeeeeee OMs 9WE
Also available as arXiv:21
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference Computer Vision (ICCV), December 2015.
Miguel Monteiro, Fabio De Sousa Ribeiro, Nick Pawlowski, Daniel C Castro, and Ben Glocker. Measuring axiomatic soundness of counterfactual image models. In The Bleventh International Conference on Learning Representations, 2022.
Norman and Justin Gilmer. Mnist-c: A robustness benchmark for computer vision. arXiv preprint arXiv: 2019.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reli- able fidelity and diversity metrics for generative models. In International conference on machine learning, pp. 7176-7185. PMLR, 2020.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models.
In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299-311 2021.
Taesung Park, Jun- Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. Advances in Neural Information Processing Systems, 33:7198-7211, 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.0065 1.64434, 2015.
Fabio De Sousa Ribeiro, Tian Xia, Miguel Monteiro, Nick Pawlowski, and Ben Glocker. High fidelity image counterfactuals with probabilistic causal models. International Conference on Machine Learning, pp. 7390-7425. PMLR, 2023.
Peter M Robinson. Root-n-consistent semiparametric regression.
Econometrica: journal of the
Econometric Society, pp. 931-954, 1988.
12

---

Published as a conference paper at ICLR 2026
Pau Rodríguez, Massimo Caccia, Alexandre Lacoste, Lee Zamparo, Issam Laradji, Laurent Charlin, and David Vazquez. Beyond Trivial Counterfactual Explanations With Diverse Valuable Expla- nations. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 1056-1065, October 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assistiee Intervention MICCAI 2015, volume 9351 of Lecture Notes in Computer Science, pp. 28. URL / 5005
2015. doi: 505. 04597
111007/978--331--4557777
Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentangle- ment of correlated factors via hausdorff factorized support. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. httpatttppp///pppeeeeiiiee wwt.tieewee.e.............. pdf ?df F4 6edad c36e 9c 6c00e1 (a..592cfdleedd.o. Published at ICLR 2023.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally Robust Neural Networks. In International Conference on Learning Representations, 2019.
Axel Sauer and Andreas Geiger. Counterfactual Generative Networks. In International Conference on Learning Representations, 2020.
Maximilian Seitzer. pytorch- fid: FID Score for Py PyTorh. http::///gitiub..commmmeetttt pytorch-fid fid. August 2020. Version 0.3.0.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In International conference on machine learning, pp. 10401-10412. PMLR, 2021.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: Hospital-sen scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106, 2017.
Wenqian Ye, Guangtao Zheng, Xu Cao, Yunsheng Ma, and Aidong Zhang, Spurious correlations in machine learning: A survey, arXiv preprint arXIw:2402.12715, 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 2023. URL hwt/httpss::/aaiv.orr/////
23C2.05545
Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah D. Goodman, and Stefano Ermon. Bias and generalization in deep generative models: An empiri- cal study. In Advances in Neural Information Processing Systems 31 (NeurIPS), pp. 10815-10824. 2018. URL htttts:://pppeeee: wiw.. html
13

---

Published as a conference paper at ICLR 2026
A
ILLUSTRATION OF PROREGEN
Fig. 6 (C).
illustrates the outline of the instantiations of ProReGen on cAVA C-GN (B), and c-DM
A.
B.
7 ===x X2 mlx1)
X2 m
7
Real
Ex
Edso
Z
Go
G
Oye
Dusus
+ or
Z
Gewr
Gan..
y
C.
Timet
Ymin:
Ymas, Y
Time
Time Embedding
Downsampling Block
Amjr
Downsampling Block
Time Embedding
hmir
Middle Block
Middle Block
Upsampling Block
Upsampling
Block
Stage-1
Stage-1l
Figure 6: Illustration of ProReGen-VARe (A), ProReGen-GAN (B), and ProReGen-DM (C).
14

---

Published as a conference paper at ICLR 2026
B ORACLE PERFORMANCE FOR EACH DATASET
We present the performance oracle classifiers for Colored-MNIST, MNIST-Coreelatio,, and Corupled-CIFARIOm in Table 2
Table 2: Test Accuracy of Oracle Classifiers for Colored-MNISST, MNIST-Correlation, and CorruptedCIIAAIO
Oracle Classifier For Digit
Average Test Accuracy
0.958
Colored-MNIST
Color
1.0
Even/Odd Digit Type
0.965
MNIST-Corrclation
Presence/Absence Zigzag
1.0
Object Type
0.849
Corrupted-CIFAR
Corruption Type
0.99
C TRAINING DATA DISTRIBUTION FOR EACH DATASET
We present the count of majority and minority training samples employed across each correlation ratio for Colored-MNIST in Table for MNIST-Correlatio in Table and for Coorupted-CIFARR 5
Table 3: Counts of Majority and Minority Samples Across Varying Levels of Correlation Strengths Explored for MNIST. For the Balanced setting, we consider equal number of samples per (digit, color) combination, Lee, around 550 samples per combination. The total number of unique combinations 100.
Correlation Strength
Minority
Majority
95%
2450
52552
98%
986
54014
99%
492
54510
99.5%
249
54751
15

---

Published as a conference paper at ICLR 2026
Table 4: Counts of Majority and Minority Samples Across Varying Levels of Correlation Strengths Explored for MNIST-Corelltion. For the Balanced setting, we consider equal number of samples per attribute combination, i.e., around 20,000 samples per combination. The total number of unique combinations is 4.
Correlation Strength
Minority
Majority
95%
2104
40000
98%
816
40000
99%
404
40000
99.5%
200
40000
Table 5: Counts of Majority and Minority Samples Across Varying Levels of Correlation Strengths Explored for Corrupted-CIFARI For the Balanced setting, consider equal number of samples attribute combination, ie.. around 800 samples per combination. The total number of unique combinations 25.
Correlation Strength
Minority
Majority
70%
6000
14000
80%
4000
16000
90%
2000
18000
99%
200
19800
Table 6: Counts of Majority and Minority Samples in natural and 94% Correlation Strengths Ex- plored for CelebA. The counts for Majority and Minority are such that (Male, Blond Hair) and (Female, Black Hair) are considered in Minority, while (Male, Black Hair) and (Female, Blond Hair) in Majority.
Correlation Strength
Minority
Majority
Natural
20170
43001
94%
2774
43001
D IMPLEMENTATION DETAILS
Network Architecture. We employ a stack of convolution and transposed convolution layers to implement the VAE (Sohn et 2015) and follow DCGAN-like architectural setup for c-GAN (Radford et al.] 2015). For causal-cHVAI and causal-GAN, we follow their official code reposito- ree: Ribeiro al. (2023) for causal-cHVAN and Kocaoglu et (2017) for causal-GAN GAN.
In both с- CAE and c-GAN, since the residual effect generator only requires adjusting residual fea- tures on images for which most the generative factors have already been produced by the major- ity DGM, the expanded layers are lightweight, comprising relatively low parameter count than the majority DGMs (Table TT]. Moreover, they are also designed such that they maintain the spatial dimension of the input image (e.g., using convolutional operations with kernel 3, stride 1, padding = 1). We present the details of their network architecture Table 10
For c-DM, we use the implementation from Nichol & Dhariwal (2021) for the naive baseline and extend it to support the weighhign bascline. Building on the foundation from Nichol & Dhari- wal (2021), we further adapt its code to incorporate
model design (shown in Fig. 6C). including feature injection and required conditioning. We consider condition integration via channel-wise con- catenation in ProReGen-DXY and maintain consistent conditioning method across all its baselines.
We present a summary of the parameter counts for the stage-I and stage-Il models of ProReGen for all considered datasets in Table [
Conditional Information 7, Residual (orthogonal) attribute = 22 mmx1) ) is broadcasted to match spatial size of the input and concatenated as additional channels provide conditional
16

---

Published as a conference paper at ICLR 2026
information the expanded layers. When have nigh-dimensional attribute 22, where only part of is predictable from 21, we predict the predictable dimensions by m(x1) to get y. As implementa- then either keep the additional (unpredictable) dimensions additional channels
tion choice, or use them to manipulate (e.g. mask) predictable dimensions. We employ the latter approach for MNIST-Correlation for ProReGen, where mask the y information using the information of line joining the two end-points of zigzag. We employ the same approach in the encoder of naively-trained conditional-Val VAE and its re-weighted version. However, employing such masking in naively-trained conditional GANN led to the model ignoring the even/odd label information, potentially due to the discriminator relying on the now easier feature, zigzag (due to masking). Therefore, we appended the coordinate information as additional channels during conditioning for conditional-GAN.
Table 7: The architecture of the expanded portion of generator network for Colored- -NIIS. The architecture of the expanded portion of discriminator or encoder is mirror of Here Cimg denotes the number of image channels and d, is the dimension of 7-
Part
Output Shape
Layer Information
Input
(B, Cimg + de, H. W)
Conv Block
(B, 64, H. W)
Conv2d(Cing + dc, 64, 3, 1, 1), GroupNorm(8, 64), ReLU Conv2d(64, 32, 3, 1), GroupNorm(8, 32), ReLU Conv2d(32, Cing. 3, 1)
Conv Block 2
(B, 32, H, W)
Output Layer
(B, Cimg. H. W)
Table 8: The architecture of the expanded portion of decoder network for MNIST-Corelltion in ProReGen- VAE. The architecture of the expanded portion of encoder is mirror of it. Here Cimg denotes the number of image channels and d, is the dimension of
Part
Output Shape
Layer Information
Input
(B, Cimg + d, H, W)
Conv Block
(B, 64, H, W)
Conv2d + de, 64, 3, 1) Conv2d(64, 64, 3, 1, 1), ReLU, Conv2d(64, 64, 3, 1, Conv2d(64, 64, 3, 1, 1), ReLU, Conv2d(64, 64, 1, I) Conv2d(64, Cing. 1, 1)
Cony Block 2
(B, 64, H, W)
Conv Block 3
(B, 64, H. W)
Output Layer
(B, Cing. H, W)
Table 9: The architecture of the expanded portion of generator network for MNIST-Correlation in ProReGen-GAN. The architecture of the expanded portion of discriminator is mirror of it. Here Cimg denotes the number of image channels and d, is the dimension of 7. This is similar to the architecture employed in ProReGen-VAE. shown in Table D with light extension to the design and the addition normalization layers to enhance the training stability.
Part
Output Shape
Layer Information
Input
(B, Cimg di H. W)
Conv Block
(B, 64, H. W)
Conv2d(Cime + di, 64, 3, 1, 1), GroupNorm(8,, 64), ReLU Conv2d(64, 64, 3, 1, 1), GroupNorm(8, 64) Conv2d(64, 32, 3, 1), GroupNorm(8, 32), ReLU Conv2d(32, 32, 3, 1, 1), GroupNorm(8, 32)
Conv Block
(B, 64, H. W)
Conv Block 3
(B, 32, H, W)
Conv Block
(B, 32, H, W)
Output Layer
(B, Cings H, W)
Conv2d(32, Cing. 3, 1, 1)
17

---

Published as a conference paper at ICLR 2026
Table 10: The architecture of the expanded portion of generator network for Corrupted- 10. The architecture of the expanded portion of discriminator is mirror of it. Here, Csmg denotes the number image channels and de denotes the dimension of and X1.
Output Shape
Layer Information
Concatenate Amir, X, along channel dim Conv2d(Ciin + 2d,, Cmiss 3, I), GroupN Ciiia); ReL.U
Residual Block
(B, Cmii, H, W)
Conv2d((mil Cmd- 3, 1, 1). GroupNorm(8. Caia)
Skip connection: Conv2d(Cims Cmid. 1, 1, 0) (or Identity channels match)
Element addition (residual connection)
Concatenate Residual Block output, 7; along channel dim
Conv2d(Cmid + 2d,, Cuaz. 3, 1), GroupNorm(8, Cmaz), ReLU
Residual Block
(B, Cmidz H, W)
Conv2d((Cmiz Cmiaz 3, GroupNorm(S. Cmia))
Skip connection: Conv2d(Cmin Cmddzs 1, 0) Identity channels match) Element-wiss addition (residual connection)
Output Layer
(B, Cimgs H, W)
Coni2d(Cmidz Cimps 3, 1, I)
Table 11: Parameter counts of Stage-I and Stage-II ProReGen for different models and datasets
Stage-1
Stage-I
Stage-11 to Stage-1
Datasets
Models
Encoder Discriminator
Decoder / Generator
Encuder Discriminator
Decoder Generator
Total
VAV
8.6M
1.6
10.2M
24.2k
27.1k
0.005
Colored-NNIST
c-GAN
674.0k
2.5M
3.2M
24.1k
27.1k
51k
0.016
c-DM
5.IM
5.1M
5.1M
5.1M
1.000
e-VAE
8.6M
166
10.2M
150k
150k
300.2K
0.030
MNIST-
Correlation
c-GAN
663.7k
2.3M
3M
67k
67k
133.6K
0.044
c-DM
5.1M
5.1M
5.1M
5.1M
1.000
Corrupted- CIFARIO
c-GAN
668.9k
239.5k
3.1M
78.6k
78.7k
157.2k
0.051
c-DM
9.9M
9.9M
9.9M
9.9M
1.000
CelebA
c-DM
23M
23M
23M
23M
1.000
E EFFECT OF TWO-STAGE TRAINING
Two-stage training of Imjr and Tres instead of single-stage training is beneficial for the quality of majority generations and overall success (correctness, FID, coverage, and density) minority gen- erations. We present visual examples of generations in Fig. 7
18

---

Published as a conference paper at ICLR 2026
Single-Stage Training
Two-Stage Training
Majority Generations
0
1
1
(
0
0
1
A
3
3
3
2
4
2
3
3
3
4
4
4
5
5
5 2
4
4
5
5
5
(a
6.
6
7
7
6
6
6
7
7
7 4
8
3
8
9
9
7
8
8
8
9
0
O
0
0
0
0 /
O
O
0
0 1
0 0
0
0
0
00
0 0
0
D
0
D
D /
0 1
0
1 / 2.
0
O
0
0
8 2 3
Minority Generations
1
/
7
1 2
/
4
2.
2
2 3
2
2
6 3333
2
2
2
2
3
J
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
8
4
9
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
9
4
4
4
7
5 6
5
5
5
45
5
5
5
5
5
5
5
5
5
5
5
5
6 6.
6
6
6
6
6
666666
6
6
4
6
6
6
6
6
6
6 7
7 7
7
7 8
7
7
7
7
7
7 7777
7
7
7
7
7
7
7
7
7 8
8
X
6
8
8
8 8 9 9
8
8
8
8
8
9 9
9
9
9
6
9 9 9
9
9
9
Figure 7: Sample majority and minority generations with two-stage training of Jmjr and Yres single-stage training for 95% correlation ratio in Colored-MNIST.
F EFFECT OF ASSUMPTION OF CAUSAL DIRECTION
We present the comparison of generated majority and corresponding minority samples when con- sidering causal direction 21 + 32 vs. 22 I1 in Fig. We consider ProReGen-GAN to present our analysis.
In Fig. B, we simply invert the causal direction, while employing the same additive formulation for unn as in our main experiments and keeping the network architecture style for Jrer consistent.
We further experiment with the general formulation Ymur Gres (mmj: (x1). x1, using: 1) the same network architecture design for Ires in Fig. B. and 2) ResNet-style architecture design for fre such that the residual operation occurs implicitly within the network, assess their potential benefit for the residual generation task with the inverted causal direction color - digit. However, no noticeable improvement was observed as shown in Fig. SC-D.
19

---

Published as a conference paper at ICLR 2026
Colored-MNIST:
X1
digit, Xg color
Minority Generation
A.
Ymar
Suir
9te (mmir(X) (x1), 7)
Real Minority Samples for Training Ires
Majority Generation Gmjr (z, x1)
yy X2
m
0
7
6
5
2
9 0
0
2
9
O
O 7
0 8
2
9
3
3
2
5
4
6
9
3
2
5
4
6
3
2
5 4
6
7
7
7
Minority Generation
B.
Y mar
Omaj
9res (Daijc (x2), ?)
Real Minority Samples for Training gno.
Majority Generation 9mj=(2 x2)
?
m
r
7
4
7
3
9
Z
4
2
3
2
9 1
7
8
7
3
4
6
8
8
8
/
5
9
?
3
4
Minority Generation (Bmjr (x2)), X2,
Real Minority Samples for Training Sive
Majority Generation 9mp: (z, Xy)
mer
7 x1
(xxx)
7
4
9
4
9
4 6
9 7
3
9
7
5 3
9
0
0
0
3
]
S
4
5
3
/
0
/
1
8
Minority Generation
9res (m (X2). xg,
Real Minority Samples for Training Ires
Majority Generation Jui- (z, xx)
(xx))
ResNet-basee architecture design for gro
6
0
4
2
4 3
8
7
O 3
9
4 0 /
3
7
d
4
1
7
7
4
9
0
5
2
3
4 7
Figure 8: Examination of the effect of causal direction on the residual generation task for ProReGen- GAN, considering 95% correlation ratio -NlOST Real minority samples used for training Ices are shown in left with corresponding majority (middle) and minority (right) generation samples. causal direction digit + color for Colored-MNIST our main experiments and present the sample results in A. We explore the effect of inverting the causal direction to color digit in B, C, and D, where the roles X1 and X2 are reversed.
We employ
Table 12: Sensitivity of ProReGen to errors in estimation of ( for GAN trained on Colored- MNIST with 95% correlation ratio
Perturbation
Correctuess
Digit Correctuess
Color Correctuess
Coverage
1
0.9256 0.0257
09490 0.0053
0.9718
0.0228
17.2562 8.86
0.7519
0.0903
0.6089 0.1216
30%
0.8909 0.0124
0.9579 0.0099
0.9291 = 0.0124
13,6764 =: 3.7443
0,8311
0.0344
0.6292 1: 0.0468
80%
0.8589 0.0139
0,9541 0.0007
0.8974 0.0117
13,4902
14889
0.839
0,0252
0.6237 1: 0.0089
G EFFECT OF ERROR IN ESTIMATION OF m(x;)
We present the sensitivity of ProReGen errors in estimation of m(x,) for GAN trained on Colored-MNISS with 95% correlation ratio in Table 12 As described in the main text, we experi- mented with three levels perturbation with increasing percentage of samples induced with random shifting in attribute MNIST with 95% correlation strength, the correctness of minority generation dropped as expected with the increase in the level of perturbation, although not rapidly and still improved over the naive model at 80% errors.
per training epoch. As shown, for FroReGen-GAN trained on Colored-
20

---

Published as a conference paper at ICLR 2026
H COMPUTATION COST
Table 13 lists the training time on Colored-MNIST with 98% correlation strength by naive, re- weighting, and ProReGen models for c-VAE, -GAN, -DM. It highlights the benefit inherent in the design choice of ProReGen: the minimal overhead of computation of stage-ll training in comparison stage-I and naive models (~ 2% for all models).
Table 13: Seconds per epoch for c-VAE, c-GAN, and c-DM for training on Colored-MNIST 98% correlation strength. The experiments were performed on single NVIDIA RTX 2080 Ti GPU with 10.75 GB VRAM and batch size 32.
Naive
Re-weighting
Stage
Stage II
VAE
8.9442 + 0.1116
8.9432 + 0.0610
8.6888 + 0.0102
0. 1863 = 0.0003
c-GAN
16.5894 + 0.0142
16.6545 1 0.0479
13,4155 + 0.0379
0.2676 = 0.0013
c-DM
121.9143 + 5.6574
125.2077 + 1,4170
125.7700 + 0.0962
2.9127 = 0.0116
21

---

Published as a conference paper at ICLR 2026
I CONVERGENCE RESULTS
Fig. provide examples of loss curves for ProReGen-VVA, GGAN, and -DM when trained on Colored-MNIIT correlation strength 95%, showing stable convergence that were typical of the experiments we observed.
ProReGen-VAE
Reconstruct on Loss
3.0000
0.35
20175
50150
0.5
80129
0.50
2000
4000
1000
2000
3000
4200
5000
6000
2000
ProReCen-GAN
Beratin
Genarater Loa
15-
20
10
2000
3003
5000
8000
7000
ProReGen-DMM
Iteration
1.0
0.
Figure Examples of loss curves ProReGen- VAE, GAN,-DM
22

---

Published as a conference paper at ICLR 2026
J PERFORMANCE SENSITIVITY TO THE SIZE OF RESIDUAL SUB-NETWORK
We present small example illustrating the sensitivity of ProReGen to the number of convolution- blocks in residual sub-net in Table [4 We can observe that both the correctness and quality of generations are worse when limiting the size of the residual sub-network to single convolution- block. Visual examples Fig. 10 further demonstrate the issue.
Table 14: An example from ProReGen-GAN trained 95% correlation strength in ColOred-MNIST
illustrating the sensitivity of performance to the size of the residual sub-network. The metric values for minority generations.
Correctness t
FID
Coverage
Density 1
Two Convolution Blocks
0.9256 0.0257
17.2562 + 7.8816
0.7519 + 0.0903
0.6089 + 0.1216
Single Convolution Block
0.6476 + 0.0536
20.6536 1.6247
0.6787 + 0.0323
0.4399 + 0.0476
Using Expansion Network with Two Convolutional--locs
Using Expansion Network with Single Convolutional-Blocks
0 0
D
O
0
0
0
O
0
Majority Color Correspondingg Digits the Remaining Colors as Their Minority
/
1
1
(
1
/
1
1
/
/
2
2
2
2
2
2
2
2
2
2
2
2
2
2-
d
2
2 3 4 5 6
3
3
3
3
3
3
3 4
3
3
3 3
3
3
3
3
3
3 4 5 6
4
4
4
4
4
4
4 5
4 4
4
4 8
4
4
4 0 6
5
5
5
S
5
$
5
5
56
5
5 6
5
5
6
6
6
6
6
6
6
6 7 8
6
6 6
6
6
6
7 7
7
7
7
7
7 9 7
7
7
7
7
7
7
8
8
e
8
8
8 9
8
8
8
8
8
8
8
8
8
9 9
9
9
9
9
9
9
9
9
9
8
9
9
4
Figure 10: Visual demonstration of the sensitivity of ProReGen to the number of convolution-blocks obtained from ProReGen-GAN trained on Colored- MNIST at 95% correlation strength and were intended to yield minority colors per row. We can observe that almost all generations match the intended color when employing two convolutional- blocks while using a single convolution-bloc yielded greater number of eπror cases.
in residual sub-net. The generations
23

---

Published as a conference paper at ICLR 2026
K
ADDITIONAL RESULTS
K.1
COLORED-MNIST
We present the comparison of FID and density metric values of ProReGen against the basclines in Fig.
Fig.
and Fig. 13
Conditional- VAE:
Naive
Re-weighting
Pseudo-supervision (causal-cHVAE)
ProReGen-VAE (Ours)
Majority
0.8
Density A
0.6
60
Bal
95%
98%
99%
99.5%
Bal
95%
98%
99%
99.5%
Minority
1.0
0.8
Density A
0.6
40
0.4
0.2
20
Bal
95%
98%
99%
99.5%
Bal
95%
98%
99%
99.5%
Correlation Strength
Figure 11: Comparison of FID and density metric values of ProReGen- VAE against the baselines for Colored-MNIST dataset.
K.2 MNIST-CORRFELATION
We present the comparison of coverage and density metric values of ProReGen against the basclines for MNIST-Correlation in Fig. and Fig. 15
24

---

Published as a conference paper at ICLR 2026
Conditional-GANN
Naive
Re-weighting
ProReGen-GAN (Ours)
Majority
150
0.8
FID
100
Denalty
50
0.4
Bal
95%
98%
99%
99.5%
95%
98%
99%
99.5%
Minority
120
100
Density A
0.6
60
40
0.2
Bal
95%
98%
99%
99.5%
Bal
95%
98%
99%
99.5%
Correlation Strength
Figure 12: Comparison of FID and density metric values of ProReGen-GAN against the baselines for Colored-MNIST dataset.
Conditional-DM:
Naive
Re-weighting
ProReGen-DM (Ours)
Majority
25
1.4
Density +
1.2
15
1.0
5
0
0.6
Bal
95%
98%
99.5%
95%
98%
99.5%
Minority
25
1.4
20
Density A
1.2
FID
15
1.0
10
0.8
0
0.6
Bal
95%
98%
99%
99.5%
Bal
95%
99%
99.5%
Correlation Strength
Figure 13: Comparison FID and density metric values of ProReGen-DM against the baselines for Colored-MNIST dataset.
25

---

Published as a conference paper at ICLR 2026
Conditionai-VAE
Naive
Re-weighting
Pseudo-supervision (causal-cHVAE)
ProReGGn--VEE (Ours)
Majority
0.5
0.5
0.4
Coverage
0.4
Density
0.2
0.2
0.1
0.1
Bal
95%
98%
99%
99.5%
Bal
95%
98%
99%
99.5%
Minority
0.6
0.5
0.5
0.4
Coverage
0.4
Density
0.3
0.3
0.2
0.1
0.1
0.0
Bal
95%
98%
99%
99.5%
Bal
95%
98%
99%
99.5%
Correlation Strength
Figure 14: Comparison of coverage and density metric values of ProReGen VAE against the base- lines for MNIST-Correlation dataset.
Conditional-GAN:
Naive
Re-weighting
Peeudo-supervision (causal-GAN)
ProReGen-GAN (Ours)
Majority
0.8
0.7
*
Coverage
0.7
Density
0.5
0.6
0.4
Bal
95%
98%
99%
99.5%
Bal
95%
98%
99%
99.5%
Minority
0.7
Denaity K.
Coverage
0.6
0.4
0.5
Bal
95%
98%
99%
99.5%
Bal
95%
98%
99%
99.5%
Correlation Strength
Figure 15: Comparison of coverage and density metric values ProReGen-GGAN against the base- lines for MNIST-Comelation dataset.
K.3
CORRUPTED-CIFARE
We present the comparison of FID and density metric values of ProReGen against the baselines for Corrupted-CIFA CIFFRRIO in Fig. [6 and Fig. I7 Due to limited oracle dataset (260 per subgroup), we
26

---

Published as a conference paper at ICLR 2026
only have 5 260 1, 300 total majority oracle samples compare against. For FID calculation (using Seitzer (020), ) we require at least 2, 500 samples. Therefore, we are unable to calculate FID for majonity generations, and present the FID metric values only for minority generations where have 20 x 260 5200 total oracle samples to compare against.
Conditional-GAN
Naive
Re-weighting
ProReGen-GAr (Ours)
Majority
Minority
1.4
1.4
Density +
1.2
1.2
1.0
1.0
0.8
0.8
Bal
70%
80%
Bal
70%
80%
Minority
100
80
60
40
20
Bal
70%
80%
Correlation Strength
Figure 16: Comparison of FID and density metric values of ProReGien-GAN against the baselines for Corrupted-CIFa CIFARRI dataset.
Conditional-DM:
Naive
Re-weighting
ProReGen-DM (Ours)
Majority
Minority
1.5
1.5
Density A
1.3
1.3
1.1
1.1
0.9
Bal
90%
99%
Bal
90%
99%
Minority
100
80
60
20
Bal
90%
99%
Correlation Strength
Figure 17: Comparison of FID and density metric values of ProReGen-DM against the baselines for Corrptte-CIFAR 10 dataset.
27

---

Published as a conference paper at ICLR 2026
K.4 CELEBA
We present additional visual results of minority image generations on CelebA with natural corre- lation (Fig- 18p and reduced proportion of female with black hair color to match that of male with blond hair color (Fig. 19).
Naive
Re-weighting
ProReGen-DM
Figure 18: Comparison of naive c-DM, reiihighting, and ProReGen-DD on minority image gener-
ation CelebA with natural correlations between gender and hair color.
28

---

Published as a conference paper at ICLR 2026
Naive
Re-weighting
ProReGen-DM
Figure 19: Comparison of naive c-DM, re-weighting, and ProReGen- DDM on minority image gener- ation on CelebA with 94% correlations between gender and hair color.
L LLM USAGE
We used the LLM tool. ChatGPT, at limited capacity. ChatGPT was leveraged for improving the quality of sentences to provide better readability and for grammatical corrections. Moreover, we utilized it to generate some portions of the graph creation scripts.
29


Reviews:
[{"review_id": "VjRpsvP9JE", "reviewer": "Reviewer_7gY6", "summary": "The paper proposes ProReGen that tackles conditional generation modeling for synthesizing under-represented minority images. ProReGen first learns the predictable part of one attribute from another and then treats what’s left as a residual signal that highlights the minority cases. Training is performed in two stages: Stage I trains a standard VAE/GAN on the common cases; Stage II freezes this backbone and adds a small residual-conditioned module trained only on minority examples to correct the missing variations. Therefore, this separates the attributes so the base learns general realism while the add-on learns rarity (minority attributes). Both VAE and GAN applicability is shown on Colored-MNIST, MNIST-Correlation, and Corrupted-CIFAR-10. The method improves correctness on rare attribute combinations and achieves competitive performance (FID, Coverage, Density)", "questions": "Please refer to the weakness section", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed approach is explained in detail and could be easy to incorporate ie (progressive training scheme and plug and play for VAE/GAN architecture).\n- Proper baselines are discussed in the related section and used to compare the performance.", "weaknesses": "- It would be great to show the results on some real and large scale vision dataset to help the readers evaluate the efficacy of the proposed approach. It’s unclear how the gains would translate to higher-resolution, richly annotated datasets (faces, scenes) or to modern diffusion/flow models, which are the latest common practice. A larger-scale study would strengthen the empirical case.\n- Stage-II adds networks and trains only on minority samples while freezing the backbone. It would help the readers to show discussions around added compute cost, convergence stability, or sensitivity to the size of the residual sub-network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6A6NYjwFKP", "reviewer": "Reviewer_dzuq", "summary": "This paper proposes a new approach for training deep generative models (DGMs) that are robust to under-represented attribute combinations. Given attributes X_1 and X_2, the model learns g(X_1, $\\gamma$) rather than g(X_1, X_2), where $\\gamma$ is the residual after predicting X_2 given X_1. This, the generative model is conditioned on orthogonal variables, rather than potentially very correlated variables. This makes the model more robust when generating rare attribute combinations.", "questions": "Can a Diffusion model be used as the underlying DGM? Does ProReGen perform well on larger, more complicated, realistic datasets? \n\nIf these points are well addressed, then I am happy to raise my score.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach of orthogonalizing attributes and the two-stage residual training approach is novel. The proposed solution is intuitive and seems very reasonable. \n\n- The fact that ProReGen is agnostic to the choice of underlying DGM, rather than being designed with only VAEs or the like, is a clear strength of the proposed approach \n\n- The writing is clear and easy to follow", "weaknesses": "- The experiments are all on toy, synthetic or partially-synthetic data. The evaluation is performed on colored-mnist, mnist correlation, and Corrupted-CIFAR10 (wherein synthetic noise was added to CIFAR images). This is not a very convincing evaluation; at least, it does not provide evidence that the proposed ProReGen approach would correctly model under-represented attribute combinations in realistic data with natural correlations. \n\n- The authors claim that the proposed approach is agnostic to the choice of deep generative models, but only discuss and implement it on VAEs and GANs. I am surprised to not see results or discussion on the case where Diffusion models (DDPM, DDIM, latent diffusion, etc) are used as the base DGM.  I don't see a technical reason why diffusion wouldn't work; the authors' claim that the choice of DGM is arbitrary seems reasonable to me.  For that reason, I think the authors should either empirically evaluate PreReGen on Diffusion models, or else clarify why this is not possible and then amended their claim about the DGM being model agnostic. \n\n- This is a minor point, but I believe the paper would benefit if it included a discussion of property-controllable VAEs [1]. PCVAE is definitely different, as it tries to learn a disentangled latent representation rather than making the model robust to correlated attributes and minority samples. But the underlying goals are similar enough that I believe comparing and contrasting the proposed approach with existing property-controllable works would be interesting. \n\n[1] Guo, Xiaojie, Yuanqi Du, and Liang Zhao. \"Property controllable variational autoencoder via invertible mutual dependence.\" ICLR. 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q65oJv5UGz", "reviewer": "Reviewer_P42c", "summary": "ProReGen is an approach for generative modeling that addresses the challenge of generating underrepresented minority samples in the presence of attribute correlations. The paper proposes a two-stage progressive learning framework inspired by Robinson's partialling-out transformation, which orthogonalizes correlated input attributes and decomposes generation into majority and residual components. The method has been evaluated on Colored-MNIST, MNIST-Correlation, and Corrupted-CIFAR10, showing improvements in minority sample generation correctness compared to naive baselines and existing mitigation strategies", "questions": "How sensitive is the performance to eros in estmaiting m(x1)?\nWhy does cGAN not perform well compared to the original paper\nWhat happens when the oracle classifiers are not so accurate\nCan the framework also incorporate continuos attributes or only discere ones?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper demonstrates both VAEs and GANs, which is diverse in nature\n- The application of partialling out transformation for generative modelling is well motivated\n- The paper evaluates across multiple metrics and provides multiple ablation studies \n- The existing methods rely on signals from an external classifier to provide pseudo-supervision to the conditional generative models. Interestingly, they partially out an image from its components", "weaknesses": "- The paper demonstrates both VAEs and GANs, which is diverse in nature\n- The application of partialling out transformation for generative modelling is well motivated\n- The paper evaluates across multiple metrics and provides multiple ablation studies \n- The existing methods rely on signals from an external classifier to provide pseudo-supervision to the conditional generative models. Interestingly, they partially out an image from its components", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "deGVZNykbb", "reviewer": "Reviewer_eCdH", "summary": "The authors study the problem of training a generative model under attribute correlation. To this end, they propose a novel approach that separates training into two stages. First, a part of the model is trained on the majority data, while in the second stage the network learns only the residual information for the correlated attribute. The efficacy of the method is evaluated on variations of MNIST and CIFAR.", "questions": "I would appreciate if the authors address the main points raised in the weaknesses section. In particular, I would encourage further experimentation on multi-attribute images (e.g., [1]). Further discussion/comparison to earlier works on disentanglement and generative modelling under imbalanced attribute distributions would also improve the paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The main idea is novel and clearly presented.\n\n- The studied problem is relevant and the proposed solution appropriately motivated.\n\n- The experimental results highlight the efficacy of the method in variations of MNIST and CIFAR10.\n\n- I appreciate the ablation of the inverse causal direction for MNIST.", "weaknesses": "- The main weakness of the paper lies in the insufficient experimental support. To showcase the effectiveness of the method, as well as the importance of the task, I would expect evaluation on more complex and natural scenarios. For example, a multi-attribute dataset e.g., [1], where this kind of attribute imbalances are naturally occurring would be appropriate.\n\n- I am missing a discussion on earlier works that study generative models under imbalanced attribute distributions. For example,  [2],  [3] to name a few.\n\n- It would be interesting to discuss how (or whether) the presented method can be applied to SOTA image generation models e.g., diffusion-based models.\n\n- On a similar note, I am missing a discussion on the form of the studied problem in scenarios, e.g., text-to image,  where the conditioning variable lives in a combinatorially large space. How would imbalances affect the performance in such models?\n\n- It would be valuable to discuss the presented method under the light of disentanglement and potentially add relevant comparisons.\n\n[1]. Deep Learning Face Attributes in the Wild\n\n[2]. Bias and generalization in deep generative models: An empirical study.\n\n[3]. Multilinear Latent Conditioning for Generating Unseen Attribute Combinations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)