SepVAE: a contrastive VAE to separate pathological patterns from healthy ones

Authors: r  louiset
e  duchesnay
a  grigis
b  dufumier
p  gori
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 09:18:59

Abstract: Contrastive Analysis VAE (CA-VAEs) is a family of Variational auto-encoders (VAEs) that aims at separating the common factors of variation between a \textit{background} dataset (BG) (\textit{i.e.,} healthy subjects) and a \textit{target} dataset (TG) (\textit{i.e.,} patients) from the ones that only exist in the target dataset. To do so, these methods separate the latent space into a set of \textbf{salient} features (\textit{i.e.,} proper to the target dataset) and a set of \textbf{common} features (\textit{i.e.,} exist in both datasets). Currently, all CA-VAEs models fail to prevent sharing of information between the latent spaces and to capture all salient factors of variation. To this end, we introduce two crucial regularization losses: a disentangling term between common and salient representations and a classification term between background and target samples in the salient space. We show a better performance than previous CA-VAEs methods on three medical applications and a natural images dataset (CelebA).

Main paper: 
 introduction: 
 One of the goals of unsupervised learning is to learn a compact, latent representation of a dataset, capturing the underlying factors of variation. Furthermore, the estimated latent dimensions should describe distinct, noticeable, and semantically meaningful variations. One way to achieve that is to use a generative model, like Variational Auto-Encoders (VAEs) (Kingma and Welling, 2013) , (Higgins et al., 2017) and disentangling methods (Higgins et al., 2017) , (Burgess et al., 2018) , (Shu et al., 2018) , (Ainsworth et al., 2018) , (Li et al., 2018) . Differently from these methods, which use a single dataset, in Contrastive Analysis (CA), researchers attempt to distinguish the latent factors that generate a target (TG) and a background (BG) dataset. Usually, it is assumed that target samples comprise additional (or modified) patterns with respect to background data. The goal is thus to estimate the common generative factors and the ones that are target-specific (or salient). For instance, consider two sets of data: 1) healthy neuro-anatomical MRIs (BG=background dataset) and 2) Alzheimer-affected patients' MRIs (TG=target dataset). As in (Jack, 2018; Antelmi et al., 2019; Dufumier et al., 2021) , given these two datasets, neuroscientists would be interested in distinguishing common factors of variations (e.g.: effects of aging, education or gender) from Alzheimer's specific markers (e.g.: temporal lobe atrophy, an increase of beta-amyloid plaques). Until recently, separating the various latent mechanisms that drive neuro-anatomical variability in neuro-degenerative disorders was considered hardly feasible. This can be attributed to the intertwining between the variability due to natural aging and the variability due to neurodegenerative disease development. The combined effects of both processes make hardly interpretable the discovery of novel bio-markers. The objective of developing such a Contrastive Analysis method would be to help separate these processes. And thus identifying correlations between neuro-biological markers and pathological symptoms. In the common features space, aging patterns should correlate with normal cognitive decline, while salient features (i.e.: Alzheimer-specific patterns) should correlate with pathological cognitive decline. 
related works: 
 Variational Auto-Encoders (VAEs) (Kingma and Welling, 2013) have advanced the field of unsupervised learning by generating new samples and capturing the underlying structure of the data onto a lower-dimensional data manifold. Disentangling methods (Higgins et al., 2017; Burgess et al., 2018; Shu et al., 2018) enable learning the underlying factors of variation in the data. While disentangling (Zheng and Sun, 2019; Chen et al., 2019 ) is a desirable property for improving the control of the image generation process and the interpretation of the latent space (Ainsworth et al., 2018; Li et al., 2018) , these methods are usually based on a single dataset, and they do not explicitly use labels or multiple datasets to effectively estimate and separate the common and salient factors of variation. Contrastive VAE (Abid and Zou, 2019; Weinberger et al., 2022; Severson et al., 2019; Ruiz et al., 2019; Zou et al., 2022; Choudhuri et al., 2019) have employed deep encoders in order to capture higher-level semantics. They usually rely on a latent space split into two parts, a common and a salient, produced by two different encoders. First methods, such as (Severson et al., 2019) , employed two decoders (common and salient) and directly sum the common and salient reconstructions in the input space. This seems to be a very strong assumption, probably wrong when working with high-dimensional and complex images. For this reason, subsequent works used a single decoder, which takes as input the concatenation of both latent spaces. Importantly, when seeking to reconstruct background inputs, the decoder is fed with the concatenation of the common part and an informationless reference vector s'. This is usually chosen to be a null vector in order to reconstruct a null (i.e., empty) image by setting the decoder's biases to 0. To fully enforce the constraints and assumptions of the underlying CA generative model, previous methods have proposed different regularizations.Here, we analyze the most important ones with their advantages and shortcomings: Minimizing background's variance in the salient space Pioneer works (Abid and Zou, 2019) have shown inconsistency between the encoding and the decoding task. While background samples are reconstructed from s', the salient encoder does not encourage the background salient latents to be equal to s'. To fix that, posterior works (Weinberger et al., 2022; Zou et al., 2022; Choudhuri et al., 2019) proposed to explicitly nullifying the background variance in the salient space. This regularization is necessary to avoid salient features explaining the background variability but not sufficient to prevent information leakage between common and salient spaces, as shown in (Weinberger et al., 2022) . Independence between common and salient spaces Only (Abid and Zou, 2019) proposed to prevent information leakage between the common and salient space by minimizing the total correlation (TC) between p(c, s|x) and p(c|x) × p(s|x). Similarly to FactorVAE (Kim and Mnih, 2019) , they used the density-ratio trick (Nguyen et al., 2010) , which requires to independently train a discriminator D λ (.) to approximate the ratio between p(c, s|x) and p(c|x) × p(s|x). However, (Abid and Zou, 2019) 's code does not use an independent optimizer for λ, which is theoretically wrong, and it thus undermines their contribution. Matching background and target common patterns Another work (Weinberger et al., 2022) , has proposed to encourage the distribution in the common space to be the same across target samples and background samples. In practice, we argue that it may encourage undesirable biases to be captured by salient factors rather than common factors. For example, suppose that we have healthy subjects (background dataset) and patients (target dataset) and that patients are composed of both young and old individuals, whereas healthy subjects are mostly old (i.e., imbalance dataset). We would expect the CA method to capture the normal aging patterns in the common space. However, forcing both p(c|x, y = 0) and p(c|x, y = 1) to follow the same distribution in the common space would probably bring to a biased distribution and thus to leakage of information between salient and common factors (i.e., aging could be considered as a salient factor of the patient dataset).This behavior is not desirable, and we believe that the statistical independence between common and salient space is a more robust property. Our contributions are three-fold:• We develop a new Contrastive Analysis method, called SepVAE, which is supported by a sound and versatile Evidence Lower BOund maximization framework.• We identify and implement two properties: the salient space discriminability and the salient/common independence, that have not been successfully addressed by previous Contrastive VAE methods.• We provide a fair comparison with other SOTA CA-VAE methods on 3 medical applications and a natural image experiment.  
contrastive variational autoencoders: 
 Let (X, Y ) = {(x i , y i )} N i=1 be a data-set of images x i associated with labels y i ∈ {0, 1}, 0 for background and 1 for target. Both background and target samples are assumed to be i.i.d. from two different and unknown distributions that depend on two latent variables: c i ∈ R Dc and s i ∈ R Ds . Our objective is to have a generative model x i ∼ p θ (x|y i , c i , s i ) so that: 1-the common latent vectors C = {c i } N i=1 should capture the common generative factors of variation between the background and target distributions and fully encode the background samples and 2-the salient latent vectors S = {s i } N i=1 should capture the distinct generative factors of variation of the target set (i.e., patterns that are only present in the target dataset and not in the background dataset). Similarly to previous works (Abid and Zou, 2019; Weinberger et al., 2022; Zou et al., 2022) , we assume the generative process: p θ (x, y, c, s) = p θ (x|c, s, y)p θ (c)p θ (s|y)p(y). Since p θ (c, s|x, y) is hard to compute in practice, we approximate it using an auxiliary parametric distribution q ϕ (c, s|x, y) and directly derive the Evidence Lower Bound of log p(x, y):EQUATIONThen, we can develop the lower bound into three terms, a conditional reconstruction term, a common space prior regularization, and a salient space prior regularization. From there, we assume the independence of the auxiliary distributions (i.e.: q ϕc,ϕs (c, s|x, y) = q ϕc (c|x)q ϕs (s|x, y)) and prior distributions (i.e.: p θ (c, s) = p θ (c)p θ (s)). Both p θ (x|y i , c i , s i ) (i.e., single decoder) and q ϕc (c|x)q ϕs (s|x, y) (i.e., two encoders) are assumed to follow a Gaussian distribution parametrized by a neural network. To reinforce the independence assumption between c and s, we introduce a Mutual Information regularization term KL(q(c, s)||q(c)q(s)). This property is desirable in order to ensure that the information is well separated between the latent spaces. Theoretically, this term is similar to the one in (Abid and Zou, 2019) . However, in (Abid and Zou, 2019) , the Mutual Information estimation and minimization are done simultaneously2 , which is theoretically wrong (see Sec. 2). Here, we correctly implement an independent optimizer to estimate the Mutual Information. To further reduce the overlap of target and common distributions on the salient space, differently from previous works, we also introduce a salient classification loss defined as E s∼q ϕs (s|x,y) log p(y|s). By combining all these losses together, we obtain the final loss L:L = -E c,s∼q ϕc,ϕs (c,s|x,y) log p θ (x|c, s, y) a) Conditional Reconstruction + KL(q(c, s)||q(c)q(s)) e) Mutual Information-E s∼q ϕs (s|x,y) log p θ (y|s)d) Salient Classification + KL(q ϕc (c|x)||p θ (c)) b) Common Prior + KL(q ϕs (s|x, y)||p θ (s|y)) c) Salient Prior(2) Conditional reconstruction The reconstruction term is -E c,s∼q ϕc,ϕs (c,s|x,y) log p θ (x|c, s, y). Given an image x (and a label y), a common and a salient latent vector can be drawn from q ϕc,ϕs with the help of the reparameterization trick. We assume that p(x|c, s, y) ∼ N (d θ ([c, ys + (1 -y)s ′ ], I), i.e: p θ (x|c, s, y) follows a Gaussian distribution parameterized by θ, centered on µ x = d θ ([c, ys + (1 -y)s ′ ]) with identity covariance matrix, and d θ is the decoder and [., .] denotes a concatenation. Therefore, by developing the reconstruction loss term, we obtain the mean squared error between the input and the reconstruction:L rec = N i=1 ||x -d θ ([c, ys + (1 -y)s ′ ])|| 2 2 .Importantly, as in (Weinberger et al., 2022; Abid and Zou, 2019) , we set the salient latent vectors of background samples to s' = 0. This choice enables isolating the background factors of variability in the common space only. Common prior Assuming p(c) ∼ N (0, I) and q ϕc (c|x) ∼ N (µ ϕ (x), σ ϕ (x, y)), the KL loss has a closed form solution, as in usual VAE. Here, both µ ϕ (x) and σ ϕ (x, y) are the outputs of the encoder e ϕc . This loss is also used in (Abid and Zou, 2019; Weinberger et al., 2022) . Salient prior First, we develop p θ (s) = y p(y)p θ (s|y), where p(y) follows a Bernoulli distribution with probability equal to 0.5. This allows us to distinguish the salient priors of background samples (p(s|y = 0)) and target samples (p(s|y = 1)). Similar to other CA-VAE methods, we assume that p(s|y = 1) ∼ N (0, I) and , as in (Zou et al., 2022) , that p(s|x, y = 0) ∼ N (s ′ , √ σ p I), with s ′ = 0 and √ σ p < 1, namely a Gaussian distribution centered on an informationless reference s ′ with a small constant variance σ p . We preferred it to a Delta function δ(s = s ′ ) (as in (Weinberger et al., 2022) ) because it eases the computation of the KL divergence (i.e., closed form) and it also means that we tolerate a small salient variation (e.g., noisy/erroneous diagnosis labels) in the background samples.The salient prior regularization encourages BG and TG salient factors to match two different Gaussian distributions centered in s ′ = 0, but with different covariance. To further reduce the overlap of target and common distributions on the salient space, we propose to minimize a Binary Cross Entropy (BCE) loss to distinguish the target from background samples in the salient space. Assuming that p(y|s) follows a Bernoulli distribution parameterized by f ξ (s), a 2-layers classification Neural Network, we obtain a BCE loss between true labels y and predicted labels ŷ = f ξ (s). This loss is not used in (Abid and Zou, 2019; Weinberger et al., 2022) .Mutual Information To promote independence between c and s, we minimize their mutual information, defined as the KL divergence between the joint distribution q(c, s) and the product of their marginals q(c)q(s). However, computing this quantity is not trivial, and it requires a few tricks to correctly estimate and minimize it. As in (Abid and Zou, 2019) , it is possible to take inspiration from FactorVAE (Kim and Mnih, 2019) , which proposes to estimate the density-ratio between a joint distribution and the product of the marginals. In our case, we seek to enforce the independence between two sets of latent variables rather than between each latent variable of a set. The density-ratio trick (Nguyen et al., 2010; Sugiyama et al., 2012) allows us to estimate the quantity inside the log in Eq.3. First, we sample from q(c, s) by randomly choosing a batch of images (x i , y i ) and drawing their latent factors [c i , s i ] from the encoders e ϕc and e ϕs . Then, we sample from q(c)q(s) by using the same batch of images where we shuffle the latent codes among images (e.g., [c 1 , s 2 ], [c 2 , s 3 ], etc.).Once we obtained samples from both distributions, we trained an independent classifier D λ ([c, s]) to discriminate the samples drawn from the two distributions by minimizing a BCE loss. The classifier is then used to approximate the ratio in the KL divergence, and we can train the encoders e ϕc and e ϕs to minimize the resulting loss:L MI = E q(c,s) log q(c, s) q(c)q(s) ≈ i ReLU log D λ ([c i , s i ]) 1 -D λ ([c i , s i ])(3) 
experiments: 
 Evaluation We evaluate the ability of SepVAE to separate common from target-specific patterns on three medical and one natural (CelebA) imaging datasets. We compare it with the only SOTA CA-VAE methods whose code is available: MM-cVAE (Weinberger et al., 2022) and ConVAE3 (Abid and Zou, 2019), using the same architecture for all models.For quantitative evaluation, we use the fact that the information about some attributes (e.g. glasses/hats in CelebA) should be present either in the common or in the salient space. Once the encoders/decoder are trained, we train a Logistic (or Linear) Regression on the estimated salient and common factors of the training set to predict the attribute presence (or value). Then, we evaluate the classification/regression model on the salient and common factors estimated from a test set. We also report the background (BG) vs target (TG) classification accuracy (Acc.) using the trained classifier for SepVAE and an independently trained classifier (still 2 layers MLPs) for the other methods.CelebA -glasses vs hat identification: In the CelebA with attributes dataset (Liu et al.) , the target set contains images of celebrities wearing glasses or hats while background images show no accessories. We used a train set of 20000 images, (10000 no accessories, 5000 glasses, 5000 hats) and an independent test set of 4000 images (2000 no accessories, 1000 glasses, 1000 hats). In Tab.1 and Fig. 3 , we demonstrate that we successfully distinguish and bacterial (1342 samples), see Fig. 7 in the Suppl. In Tab. 2, we demonstrate that our method can produce a salient space that captures the pathological variability, as it better distinguishes the two subgroups. Ablation: In Tab. 2, we also propose to disable different components of the loss to show that the proposed full model is always better on average. no MI means that we removed the Mutual Information loss. no CLSF means that we disabled the Salient Classification loss. no SAL means that we ignored the Salient Prior loss. Parsing neuro-anatomical variability in psychiatric diseases Given a background population of Healthy Controls (HC) and a target population suffering from a Mental Disorder (MD), the objective is to capture the pathological factors of variability in the salient space, such as psychiatric and cognitive clinical scores, while isolating in the common space the patterns related to demographic variables, such as age and sex, or acquisition sites. For each experiment, we gather T1w anatomical VBM (Ashburner and Friston, 2000) preprocessed images of HC and MD subjects. We divide them into 5 TRAIN, VAL splits (0.75, 0.25) and evaluate in a cross-validation scheme the performance of SOTA CA-VAEs. Schizophrenia: We merged images of schizophrenic patients (TG) and healthy controls (BG) from the datasets SCHIZCONNECT-VIP (Wang et al., 2016) and BSNIP (Tamminga  
conclusions and perspectives: 
 Building onto Contrastive Analysis methods, we discuss previously proposed regularizations about (1) the matching of target and background distributions in the common space and(2) the overlapping of target and background priors in the salient space. These regularizations may fail to prevent information leakage between common and salient spaces. We thus propose two alternative solutions: salient discrimination between target and background samples, and mutual information minimization between common and salient spaces. We demonstrate superior performances on radiological and two neuro-psychiatric applications, where we successfully separate the pathological information of interest (diagnosis, pathological scores) from the "nuisance" common variations (e.g., age, site). The development of CA methods offers a large spectrum of perspectives. It could be further extended to multiple target datasets (healthy Vs several pathologies) and to other generative models, such as GANs or Diffusion Models, for improved generation quality. Furthermore, generative models could also be coupled with contrastive learning losses, to improve the representation quality (Dufumier et al., 2023) .Figure 5 : Illustration of the regularization loss within the salient space. As in MM-cVAE, the prior q ϕs(s|x,y=0) ∼ s' on the background samples (blue) forces their variance to be as small as possible. However, as the prior on target samples (green) follow a normal distribution, they may overlap with the background distribution. To avoid this case, our method trains a non-linear classifier to avoid the overlap of both distributions with a margin proportional to σ q .Figure 6 : CelebA accessories dataset. We used a train set of 20000 images (10000 no accessories, 5000 glasses, 5000 hats) and an independent test set of 4000 images (2000 no accessories, 1000 glasses, 1000 hats) and ran the experiment 5 times to account for initialization uncertainty. Images were centered on the face and then resized to 64 × 64, pixels were normalized between 0 and 1.optimizer. Oddly we found that re-instantiating it at each epoch led to better results (for concurrent works also), we think that it is because it forgets momentum internal states between the epochs. The models were trained during 250 epochs. To note, in the original contribution, MM-cVAE used latent spaces of 16 for the salient space and 6 for the common space and a different architecture but we noticed that it led to artifacts in the reconstruction (see (Weinberger et al., 2022) ). Also, we did not succeed in reproducing their performances with their code, their model, and their latent spaces, even with the same experimental setup. We, therefore, used our model setting which led to better performances across each method with batch size equal to 512. We used β c = 0.5 and β s = 0.5, κ = 2, γ = 1e -10, σ p = 0.025. For MM-cVAE we used the same learning rate, β c = 0.5 and β s = 0.5, the background salient regularization weight 100, common regularization weight of 1000. Original dataset image description from (et al., 2018) . The dataset is available at https: //www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia. 
 
 Appendix: Train set images were graded by 2 radiologist experts and the independent test set was graded by a third expert, the experiment was run 5 times to account for initialization uncertainty. Radiographies were selected from a cohort of pediatric patients aged between one and five years old from Guangzhou Women and Children's Medical Center, Guangzhou. TRAIN set images were graded by 2 radiologists experts and the independent TEST set was graded by a third expert to account for label uncertainty. Images are of size 64 × 64, pixel were normalized between 0 and 1. For this experiment, we use a standard encoder architecture composed of 4 convolutions (channels 3, 32, 32, 32, 256) , kernel size 4, and padding (1, 1, 1, 0). Then, for each mean and standard deviations predicted (common and salient) we used two linear layers going from 256 to hidden size 256 to (common and salient) latent space size 128. The decoder was set in a symmetrical manner. We used the same architecture across all the concurrent works we evaluated. We used a common and latent space dimension of 128 each. The learning rate was set to 0.001 with an Adam optimizer.Oddly we found that re-instantiating it at each epoch led to better results (for concurrent works also), we think that it is because it forgets momentum internal states between the epochs. The models were trained during 100 epochs with batch size equal to 512. We used β c = 0.5 and β s = 0.1, κ = 2, γ = 5e -10, σ p = 0.05. For MM-cVAE, we used the same learning rate, β c = 0.5 and β s = 0.1, the background salient regularization weight 100, common regularization weight of 1000.The task of identifying consistent correlations between neuro-anatomical biomarkers and observed symptoms in psychiatric diseases is important for developing more precise treatment options. Separating the different latent mechanisms that drive neuro-anatomical variability in psychiatric disorders is a challenging task. Contrastive Analysis (CA) methods such as ours have the potential to identify and separate healthy from pathological neuro-anatomical patterns in structural MRIs. This ability could be a key component to push forward the understanding of the mechanisms that underlie the development of psychiatric diseases. As explained in the main text, given a background population of Healthy Controls (HC) and a target population suffering from a Mental Disorder (MD), the objective is to capture the pathological factors of variability in the salient space, such as psychiatric and cognitive clinical scores, while isolating the patterns related to demographic variables, such as age and sex, or acquisition sites to the common space. For each experiment, we gather T1w anatomical VBM (Ashburner and Friston, 2000) pre-processed images of HC and MD subjects of size 128 × 128 × 128. We divide them into 5 TRAIN, VAL splits (0.75, 0.25) and evaluate in a cross-validation scheme the performance of SOTA CA-VAEs. Let us note that this is a challenging problem, especially due to the high dimensionality of the input and the scarcity of the data. Notably, the measures of psychiatric and cognitive clinical scores are only available for some patients, making it scarce and precious information.Images are of size 128 × 128 × 128 with voxels normalized on a Gaussian distribution per image. Experiments were run 3 times with a different train/val/test split to account for initialization and data uncertainty. For this experiment, we use a standard encoder architecture composed of 5 3D-convolutions (channels 1, 32, 64, 128), kernel size 3, stride 2, and padding 1 followed by batch normalization layers. Then, for each mean and standard deviations predicted (common and salient), we used two linear layers going from 32768 to hidden size 2048 to (common and salient) latent space size 128. The decoder was set symmetrically, except that it has four transposed convolutions (channels 128, 64, 32, 16, 1) , kernel size 3, stride 2, and padding 1 followed by batch normalization layers. We used the same architecture across all the concurrent works we evaluated. We used a common and latent space dimension of 128 each. The models were trained during 51 epochs with a batch size equal to 32 with an Adam optimizer. For the Schizophrenia experiment, for Sep VAE, we used a learning rate of 0.00005, β c = 1 and β s = 0.1, κ = 10, γ = 1e -8, α = 1 0.01 . For MM-cVAE we used the same learning rate, β c = 1 and β s = 0.1, the background salient regularization weight 100, common regularization weight of 1000. For the Autism disorder experiment, we used a learning rate of 0.00002, β c = 1 and β s = 0.1, κ = 10, γ = 1e -8, σ p = 0.01. For MM-cVAE we used the same learning rate, β c = 1 and β s = 0.1, the background salient regularization weight 100, common regularization weight of 1000. images where we shuffle the latent codes among images (e.g., [c 1 , s 2 ], [c 2 , s 3 ], etc.). Once we obtained samples from both distributions, we trained an independent classifier D λ ([c, s] ) to discriminate the samples drawn from the two distributions by minimizing a BCE loss. The classifier is then used to approximate the ratio in the KL divergence, and we can train the encoders e ϕc and e ϕs to minimize the resulting loss:L MI = E q(c,s) log q(c, s) q(c)q(s)EQUATION)where the ReLU function forces the estimate of the KL divergence to be positive, thus avoiding to back-propagate wrong estimates of the density ratio due to the simultaneous training of D λ ([c, s]). Contrarily to (Abid and Zou, 2019) , it is important to use an independent optimizer for D λ to ensure that the density ratio is well estimated. The pseudo-code is available in Alg. 1, and a visual explanation is shown in Fig. 8 . In(Abid and Zou, 2019), Alg. 1 suggests that the MI estimation and minimization depend on two distinct parameter updates. However, in their code, a single optimizer is used. Moreover, in Sec. 3, authors write: "discriminator is trained simultaneously with the encoder and decoder".ConVAE implemented with our MI minimization, i.e.: with independently trained discriminator.Our evaluation process is different from(Weinberger et al., 2022) as their TEST set has been used during the model training. Here, TRAIN and TEST are correctly separated. 


Decision: The paper introduces SepVAE, a Contrastive Analysis Variational Auto-Encoder (CA-VAE) variant designed to distinguish between common and salient features in datasets, particularly useful in medical imaging. SepVAE introduces new regularization losses to enhance this separation, improving upon existing CA-VAE models. The performance is evaluated on medical datasets and CelebA, demonstrating superiority over baselines. Reviewers generally praise the paper's organization, innovation, and performance but highlight some limitations and suggest improvements.

Despite some limitations, the reviewers generally recommend accepting the paper with minor revisions. Suggestions include expanding the evaluation to other medical datasets, conducting ablation studies to clarify the impact of the new loss terms, and including experiments with diagnostic algorithms to demonstrate practical utility. Addressing these suggestions would enhance the paper's contribution and make it more compelling for publication.

Reviews: review: paper_summary: paper_summary: Paper Summary: The paper presents SepVAE, a novel approach within the family of Contrastive Analysis Variational Auto-Encoders (CA-VAEs), which aims to distinguish between common factors of variation in a background dataset (typically representing healthy subjects) and unique factors in a target dataset (typically representing patients). The main innovation of SepVAE lies in its introduction of two new regularization losses to improve the separation between common and salient (target-specific) features in the latent space: a disentangling term that differentiates between common and salient representations, and a classification term that discriminates between background and target samples in the salient space. According to the authors, SepVAE outperforms existing CA-VAE models in separating pathological patterns from healthy ones across three medical applications and a natural image dataset (CelebA).

Strength: SepVAE introduces novel regularization losses to better separate common and salient features between datasets, addressing a significant limitation in existing CA-VAEs. The performance is also outstanding comparing to existing baselines. The paper is in general well-organized with no English grammar errors.

Weakness: While the paper demonstrates improvements over existing methods, the evaluation is somewhat limited to specific types of medical data and the CelebA dataset, which might not cover all potential use cases or data types. Will the authors consider implementing the model on other medical image dataset? Another suggestion is that, an ablation study may be helpful here. The authors may consider removing some structures in the SepVAE model (such as the novel regularization losses) and compare the corresponding results. In this case people can better understand where the efficiency of this model comes from.

In conclusion, this is a typical application of deep learning techniques to medical imaging. Although there are some weakness as mentioned above, this is a fairly successful paper. So, I will suggest to accept this paper after minor revision. The authors may want to address the weakness I mentioned in above.
main_review: main_review: In conclusion, this is a typical application of deep learning techniques to medical imaging. Although there are some weakness as mentioned above, this is a fairly successful paper. So, I will suggest to accept this paper after minor revision. The authors may want to address the weakness I mentioned in above.
strength_weakness: strength_weakness: Weakness: While the paper demonstrates improvements over existing methods, the evaluation is somewhat limited to specific types of medical data and the CelebA dataset, which might not cover all potential use cases or data types. Will the authors consider implementing the model on other medical image dataset? Another suggestion is that, an ablation study may be helpful here. The authors may consider removing some structures in the SepVAE model (such as the novel regularization losses) and compare the corresponding results. In this case people can better understand where the efficiency of this model comes from.
questions: questions: Not applicable.
paper_summary: paper_summary: The authors introduce a novel Contrastive Variation Autoencoder (CA-VAE) model equipped with two new regularization losses: disentangling between common and salient representations, and classification between background and target samples in the salient space. This enhancement addresses limitations in existing CA-VAE models, such as information sharing between latent spaces and incomplete capture of salient factors of variation. The performance of the proposed model is demonstrated on three medical datasets and the CelebA dataset.
main_review: main_review: I would like to thank the authors for reponding the review comments. The authors, by and large, replied to my comments. After the reviews, my score is more positive but still not at the level of strong accept. Therefore I will keep my score as it is.
strength_weakness: strength_weakness: The specific term in the loss function responsible for discriminating between subgroups is not clearly identified or explained.

Robustness against data imbalance is not justified

The paper lacks experimental results demonstrating the use of the extracted features to train a competitive diagnostic algorithm, such as a diagnostic classification model.
questions: questions: It is unclear from the paper which specific term in the loss function is responsible for discrimination between subgroups. It's uncertain whether this discrimination arises as a by-product of the other losses. 

It would be beneficial to provide evidence demonstrating the model's robustness against data imbalance.

The paper lacks clarity on how competitive the extracted features are when applied to a downstream task. Specifically, it is unclear whether these features enable the training of diagnostic models that perform at the level of state-of-the-art algorithms reported in the literature.
paper_summary: paper_summary: This work introduces a  Contrastive Analysis method with which we estimate the common generative factors and the ones that are
target-specific. The motivation for his lies in distinguishing variations like aging, gender from markers which are Alzheimer’s specific like temporal lobe atrophy
main_review: main_review: The reviewers have addressed my comments and I keep my initial rating of accept as the novelty of the work lies in implementing salient space discriminability and the salient/common independence, and SepVAE has also been well experimented and ablated through experiments.
strength_weakness: strength_weakness: While the proposed method definitely has improvements over the previous methods, it is not clear to me how exactly the introduced loss terms in the proposed loss is helpful in the final performance. I understand that Table 2 tries to show this, the authors basically remove each individual loss terms to actually show how much each loss terms influence. I think it might be better if the authors show how adding each individual term actually affects the performance and so we can also understand how much of an overlap is present between the loss in the previous methods ConVAE MM-VAE.
questions: questions: It would be great to have more details on how exactly the work is different from the previous VAE methods proposed to differentiate. The writing can be changed such that the focus is more on how exactly this method is different from the conVAE based previous methods. I understand that the authors try to do this in the end of Related Works section but I do not feel it is sufficient.

Comments: title: Response to R8ZKa's second point.
Response to RzXYa's third point
Response to RzXYa's second point
Response to RD7R9
Response to RzXYa's first point
Response to R8ZKa's first point.
comment: I thank the reviewers for their comments and believe the papers deserves to be accepted to MIDL.
''Another suggestion is that an ablation study may be helpful here. The authors may consider removing some structures in the SepVAE model (such as the novel regularization losses) and comparing the corresponding results so that people can better understand where the efficiency of this model comes from.'' 


Answer:  We provided an ablation study in Table. 2 in the original manuscript. As suggested by Reviewer D7R9, we enrich it by providing the three remaining loss combinations (MI + SAL, MI + CLSF, and MI + SAL + CLSF). We report the new results in the following table. Please note that ConVAE is equivalent to SepVAE without the CLSF and the SAL losses. We clarify that in the revised version of the manuscript writing "\textit{no SAL means that we ignored the informationless constraint on the Salient Prior loss for background samples (as in Abid and Zou)}."

|                                           | Subgrp Acc                 | Subgrp Acc                 | Bg vs Tg Acc            | Bg vs Tg Acc               |
|-------------------------------------------|----------------------------|----------------------------|-------------------------|----------------------------|
|                                           | salient $\uparrow$         | common $\downarrow$        | salient $\uparrow$      | common $\downarrow$        |
| ConVAE (:= SepVAE no \textbf{CLSF + SAL}) | 82.30$\pm$1.53             | 73.58$\pm$1.84             | 67.80$\pm$5.93          | 58.05$\pm$7.17             |
| MM-cVAE                                   | 82.86$\pm$1.87             | 74.35$\pm$3.19             | 70.44$\pm$2.69          | 59.94$\pm$5.88             |
| SepVAE                                    | \textbf{84.78$\pm$0.42}    | \textbf{70.92$\pm$1.39}    | \textbf{78.13$\pm$3.03} | 57.52$\pm$4.14             |
| SepVAE no \textbf{MI}                     | 84.10$\pm$0.48             | 71.79$\pm$2.94             | 75.18$\pm$5.69          | 60.35$\pm$4.73             |
| SepVAE no \textbf{CLSF}                   | \underline{84.71$\pm$1.19} | 73.58$\pm$2.19             | 71.91$\pm$4.65          | \underline{55.79$\pm$5.41} |
| SepVAE no \textbf{SAL}                    | 83.98$\pm$0.85             | 72.61$\pm$2.05             | 73.03$\pm$2.97          | 61.43$\pm$2.25             |
| SepVAE no \textbf{MI + SAL}               | 81.58$\pm$3.68             | \underline{71.73$\pm$5.17} | 61.24$\pm$3.89          | \textbf{54.33$\pm$5.30}    |
| SepVAE no \textbf{MI + CLSF}              | 84.25$\pm$0.47             | 73.17$\pm$3.15             | 53.10$\pm$1.63          | 57.58$\pm$6.74             |
| SepVAE no \textbf{MI + SAL + CLSF}        | 81.78$\pm$2.12             | 76.71$\pm$2.10             | 62.87$\pm$7.15          | 59.37$\pm$5.69             |
''The paper lacks clarity on how competitive the extracted features are when applied to a downstream task. Specifically, it is unclear whether these features enable the training of diagnostic models that perform at the level of state-of-the-art algorithms reported in the literature.''


Answer: First of all, we can observe that we reported the diagnosis classification scores in the salient space and the common space in all our experiments ("BG vs TG").

The salient latent space reaches an AUC of around $79.15\pm3.39$ \% in discriminating schizophrenia individuals from healthy controls, which is close to what has been reported in the literature (from 79\% to 81\% in [A]) with deeper convolutional architectures. Similarly, the salient latent space reaches an AUC of around $59.73\pm1.78$ \% in discriminating autistic individuals from healthy controls, which is close to what has been reported in the literature (from 59.5\% to 61\% in [A]) with deeper convolutional architectures.

On BRATS-2021, SepVAE reaches a balanced accuracy of $99.31$ \%, which is close to what has been reported in the literature (as [B] reported a classification result of $99.98$\% with a pre-trained Inception-ResnetV2 neural network, and [C] reported a classification result of $99.29$).

[A] B. Dufumier, Representation learning in neuroimaging
Transferring from big healthy data to small clinical cohorts, Thesis Manuscript.

[B] D. S. Islam, R. Udhayakumar, S. Kar, P. N, U. S. Aswal and D. K. J. B. Saini, "Enhancing Brain Tumor Detection Classification by Computational Intelligence," 2023 Second International Conference on Electronics and Renewable Systems (ICEARS), Tuticorin, India, 2023, pp. 1044-1049.

[C] G. Bompem, D. Pandluri, Batch Normalization Based Convolutional Neural Network for Segmentation and Classification of Brain Tumor MRI Images, INASS 2023.
"It would be beneficial to provide evidence demonstrating the model's robustness against data imbalance." 

Answer: In the paper, we pinpointed data imbalance as a possible example of data biases. We argued that forcing the distribution in the common space to be the same across target and background samples may undermine the capture of common factors in the common space, since some of them might be actually put in the salient latent space by the method.

We have not used a highly imbalanced dataset to show that behavior but the effect of data biases can still be observed in the neuropsychiatric experiments. Indeed, in these experiments, the age distribution differs between healthy controls and diseased individuals in the schizophrenia disorder dataset, as shown in [A], Table 2.1 (SCHIZCONNECT and BSNIP), which can thus be considered as a data bias. As shown in the lower table, 
matching the healthy and diseased sample distributions in the common space undermines the capture of patterns associated with the age in the common space of MM-cVAE, but not in SepVAE and ConVAE. Thus, as the reconstruction objective requires the input information to be preserved in the latent space (either common or salient), age-related patterns naturally emerge in the salient space, even if they should be in the common latent space. We will add this Table and discussion in the Supplementary of the revised version of the manuscript.

|         | Age MAE S$\uparrow$    | Age MAE C $\downarrow$    |
|---------|------------------------|---------------------------|
| ConVAE  | \underline{7.46$\pm$0.18}          | \underline{6.40$\pm$0.26} |
| MM-cVAE | 7.10$\pm$0.34          | 6.55$\pm$0.18             |
| SepVAE  | \textbf{7.98$\pm$0.25} | \textbf{6.40$\pm$0.13}    |

[A] B. Dufumier, Representation learning in neuroimaging
Transferring from big healthy data to small clinical cohorts, Thesis Manuscript.
1) ''I think it might be better if the authors show how adding each individual term actually affects the performance and so we can also understand how much of an overlap is present between the loss in the previous methods ConVAE MM-VAE'' \\

Answer:  As suggested by the Reviewer, we added the missing ablation combinations (MI + SAL, MI + CLSF, and MI + SAL + CLSF) in the Ablation Study of the original paper (Table. 2) and specified which combination refers to the previous method ConVAE.


2) ''It would be great to have more details on how exactly the work is different from the previous VAE methods proposed to differentiate. The writing can be changed such that the focus is more on how exactly this method is different from the conVAE-based previous methods. I understand that the authors try to do this at the end of the Related Works section but I do not feel it is sufficient.''

Answer: We have modified the revised version of the manuscript to better clarify the differences between our method and the previous works. Changes have been highlighted in blue.
''It is unclear from the paper which specific term in the loss function is responsible for discrimination between subgroups. It's uncertain whether this discrimination arises as a by-product of the other losses.''

Answer: The unsupervised discrimination between the subgroups arises from \textit{all} the proposed regularization losses which participate, as shown in the ablation study, to correctly separate the common from the salient information. Indeed, the discriminative information about the subgroups should only be in the salient space and not in the common space. Being an \textit{unsupervised} validation, there is no knowledge or information about the subgroups during training.
''The evaluation is somewhat limited to specific types of medical data and the CelebA dataset, which might not cover all potential use cases or data types. Will the authors consider implementing the model on other medical image datasets?''

This is an interesting point. In this article, we have focused on \textit{anatomical} imaging data (T1w MRI and X-Ray) providing quantitative and qualitative results on three medical applications and one natural images dataset (CelebA). We agree with the Reviewer that extending the proposed method to other anatomical imaging modalities, such as diffusion MRI or histopathology, or even to functional imaging data, like fMRI or EEG, would be of interest and we are currently working on it. Another interesting extension would be the integration of non-imaging modalities (e.g., clinical, genetic data). Even if the methodological framework is rather generic and could be easily extended to such new modalities, the encoder-decoder architecture should be probably changed and adapted to the modality under analysis. This is left as future work.
We thank the Reviewers for their thoughtful and useful feedback. We have provided our response to each of the points raised. Based on the received reviews, we have updated our manuscript (changes highlighted in blue).