A Mutual Information Perspective on Multiple Latent Variable Generative Models for Positive View Generation
Authors: Dario Serez
Marco Cristani
Alessio Del Bue
Vittorio Murino
Pietro Morerio
Venue: Accepted by TMLR ()

Abstract: In image generation, Multiple Latent Variable Generative Models (MLVGMs) employ multiple latent variables to gradually shape the final images, from global characteristics to finer and local details (e.g., StyleGAN, NVAE), emerging as powerful tools for diverse applications. Yet their generative dynamics remain only empirically observed, without a systematic understanding of each latent variable's impact. In this work, we propose a novel framework that quantifies the contribution of each latent variable using Mutual Information (MI) as a metric. Our analysis reveals that current MLVGMs often underutilize some latent variables, and provides actionable insights for their use in downstream applications.

With this foundation, we introduce a method for generating synthetic data for Self-Supervised Contrastive Representation Learning (SSCRL). By leveraging the hierarchical and disentangled variables of MLVGMs, our approach produces diverse and semantically meaningful views without the need for real image data.

Additionally, we introduce a Continuous Sampling (CS) strategy, where the generator dynamically creates new samples during SSCRL training, greatly increasing data variability. Our comprehensive experiments demonstrate the effectiveness of these contributions, showing that MLVGMs' generated views compete on par with or even surpass views generated from real data. 

This work establishes a principled approach to understanding and exploiting MLVGMs, advancing both generative modeling and self-supervised learning.  Code and pre-trained models at: https://github.com/SerezD/mi_ml_gen

Published in Transactions on Machine Learning Research (09/2025)
A Mutual Information Perspective on Multiple Latent Variable Generative Models for Positive View Generation
Dario Serez
dario.serez@hit..k it
IstStuto Italiano di Tecnologia, Italy
Marco Cristani
mavoo.cristami@hninn
Reykjavik University, Iceland University of Verona, Haly
Alessio Del Bue
alessio. delbue
Istiudo Italino di Tecnologia, Italy
Vittorio Murino
nittorio.marinoo Wit.
Istituto Habinoo di Tecnologia, Raly University of Verona, Italy
Pietro Morerio
Metro.morerio@il..
Istituto Italiono di Tecnologia, Italy
Reviewed on OpenReview: https: //tp://ppppe openreview. net/ forum? id= waj2222PP
Abstract
In image generation, Multiple Latent Variable Generative Models (MIVGMs) employ multiple latent variables to gradually shape the final images, from global characteristies to finer and local details (e.g., StyleGAN, NVAE), emerging powerful tools for diverse applications. Yet their generative dynamics remain only empirically observed, without a systematic understanding of each latent variable's impact. In this work, we propose a novel framework that quantifies the contribution of each latent variable using Mutual Information (MI) as a metric. Our analysis reveals that current MIVGMs often underutilize some latent variables, and provides actionable insights for their use in downstream applications.
With this foundation, we introduce a method for generating synthetic data for Self-Supervised Contrastive Representation Learning (SSCRL). By leveraging the hierarchical and disen- tangled variables of MLVGMs, our approach produces diverse and semantically meaningful views without the need for real image data. Additionally, introduce Contimuous Sam- pling (CS) strategy, where the generator dynamically creates new samples during SSCRL training, greatly increasing data variability. Our comprehensive experiments demonstrate the effectiveness of these contributions, showing that MLVGMs' generated views compete on par with or even surpass views generated from real data.
This work establishes principled approach to understanding and exploiting MLVGMs, advancing both generative modeling and self-supervised learning. Code and pre-trained models at: htttps///ithhbb.oo /Sa////rrDDDim ml gon
1 Introduction
Latent Variable Generative Models (LVGMs), such as Variational Autoencoders (VAEs) (Kingma & Welling 2014 Rezende et al. 2014) and Generative Adversarial Networks (GANs) (Goodfellow al.) 2014), are foundational approaches for image generation. Given a random variable X X, representing high-dimensional

---

Published in Transactions on Machine Learning Research (09/2023)
layer 1
layer? 2
layer 3
(b)
Figure I: (a) Multiple Latent Variable Generative Models utilize multiple latent variables (here Z1, Z2, Za), which are sequentially introduced at different layers of the generative network produce high-quality images. (b) The base image (left), generated using points 21, 22, Z3, can be selectively modified by altering individual latents (z) to 21, 22 to 22, or 23 to 23). Each latent affects the final image differently, at first influencing broader, global attributes and later refining finer, local details (darker to lighter shading in the figure).
pictures pixel space, LVGMs aim to approximate the underlying data distribution p(X). To achieve this, they learn paramcterized generator g(z; 0) = x, where Z E denotes a latent variable sampled from a simpler and known distribution in the latent. space. A key objective of the learning process is to ensure that the generator is continuous, such that neighboring latent points z' and z" are mapped to perceptually similar outputs x' and x". This regularization of the latent space allows LVGMs to generate novel content. and meaningfully interpolate latent features (Radford et al. 2016 Higgins et al. 2017).
Over the years, advancements in latent generative modeling have focused on the use of multiple latent
variables, rather than a single latent code (Vahdat & Kautz, 2020 Karras et al. 2019 2020 2021 Sauer
et al.) 2022). By incorporating latent variables different layers of the network (see Figure
these
Multiple Latent Variable Generative Models (MTVGMs) offer hierarchical structure where early latent. codes influence broad, global features and later codes refine finer, local details (Figure [b]. The resulting architecture enhances control over the image synthesis process, enabling the generation of high-resolution images with richer detail and improved precision.
The ability to disentangle global and local features in image generation not only improves the visual quality of generated images but also widens the application scope of these models. For instance, the StyleGAN architecture (Karras et al. 2019) has demonstrated exceptional performance in image editing (Alaluf et al,
2022 Pehlivan et al.) 2023 manipulation (Tov et al.) 2021). and translation (Richardson et al. 2021 Additionally, recent studies have shown that MLVGMs can serve effective foundation models for tasks such as adversarial purification (Serez et al.
2025 Collectively, these findings highlight the versatility of MLVGMs, showcasing their utility not only in creative and generative domains but also as pre-trained models for broader applications.
Nevertheless, existing research primarily leverages the "global-to-local" behavior of MIVGMs as an empirical tool, applying it across diverse tasks without delving into the mechanics of latent variable utilization. In other terms, these approaches assume that earlier latent variables shape coarse image attributes while later ones refine fine details, but they do so without formally analyzing how each latent variable contributes to image generation. As result, the internal dynamics of MLVGMs remain poorly understood.
To address this gap, we propose a novel approach that establishes a direct relationship between feature distances each latent space (Z,, Zg,... ... Z,) and mutual information (MT) shifts the shared image space X. The key insight is that producing an oquivalent MI shift in the output space requires incrcasingly larger perturbations (4:) we move deeper into the generative hierarchy i.e. from Zi to Zn This quantitatively confirms the global-to-local pattern and reveals how influence diminishes across successive latent variables (sce Figure 2
By grounding this analysis in information theory, our approach moves beyond intuition and provides a principled framework for understanding latent variable roles MLVGMs. It enables direct comparisons between latent spaces and exposes inefficiencies in how models allocate representational power. Notably, we reveal the underutilization of later-stage latents in all tested models, opening to further architectural improvements and enabling better strategies for leveraging MLVGMs in downstream tasks.
2

---

Published in Transactions on Machine Learning Research (09/2025)
Multiple Latent Spaces
Image Space
x gi2, Zy, Z)
Z
Z
X
g g(Z,, Zz Z2,62)
1xxx
g(Z,, Zo, Z3.0)
f(X,x))
Z3
23
X Zi Zo. Z,,00)
Figure 2: Illustration of our findings on the "global-to-local" behavior in MLVGMs. On the left, perturbations are applied to each latent variable independently (Z1 - 21, Z2 + 22, and Z₂ - 2s), with the average perturbation magnitude increasing across latent spaces (μ1 < jr2 < (a)). On the right, each perturbed latent. variable is used to generate modified images (X', X", and X"" from the original X. Notably, the increasing perturbation magnitude in the latent space maintains approximately equal Mutual Information shifts in the image space: I(X:X) x I(X;X" se X"): This provides the first quantitative measure of the "global-to-local" property, where carlier latents affect global features and later latents refine local details.
With this understanding, we propose novel application of MIVGMs in Self-Supervised Contrastive Rep- resentation Learning (SSCRL). In SSCRL, feature extractors, or encoders f(x; 6) with parameters é, are trained to represent data by contrasting positive and negative views. Positive views are semantically similar images, encouraged to have close representations in the latent space, while negative views correspond to unrelated data points that are forced to have distant representations. Therefore, we propose to leverage the different impacts of multiple latent variables in MLVGMs to manipulate specific features and generate positive views. This approach enables the training of SSCRL encoders without relying on real data, demonstrating the potential of MLVGMs as pre-trained models for producing high-quality synthetic images tailored for reprosentation learning.
The primary objective of SSCRL is to enforce desired set of invariances in the learned representations (Xiao et al.) 2020), achieved by creating valid positive views. Figure 3 compares the proposed method with standard pixcl-space augmentations and single latent variable generative models (LVGMs) for view generation. the typical approach (Figure [aj, a finite set of hand-crafted transformations, such as color adjustments, cropping, or flipping, applied directly in the pixel space. Alternatively, invariances can be introduced at the latent level of a pro-trained LVGM (Figure 3bp. However, in LVGMs, all image features are entangled within single latent space, making it difficult generate specific invariances (e.g., altering fur patterns) without inadvertently affecting global features, such as changing the dog breed (e.g., from Australian Terrier to Yorkshire Terrier in the figure). In contrast, MLVGMs inherently disentangle global and local features, enabling precise control over specific characteristics. For instance, using MLVGMs, attributes like fur patterns or color can be modified while preserving global features, such as the dog breed. This is achieved by independently perturbing cach latent variable to a dosired magnitude, as illustrated in Figure 3c
The use of generative models create both anchor and positive views introduces significant challenge: the lower classification accuracy typically observed when training synthetic data compared to real data RRavuri& & Vinyals 2019). Prior studics, such as Besnier et al. (2020); Lampis et al. (2023). have identified the lack of diversity in generated images as a primary factor contributing to this issue. To mitigate this, these works propose increasing dataset diversity by sampling and storing a large number syuthetic images before training. In contrast, we propose novel approach callod Continuous Sampling (CS) to address this limitation. With CS, new images are generated "online" during the encoder network's training process, offering three key ad advantages (i) need to store large quantities of synthetic data, thereby preserving disk space; (ii) no data
3

---

Published in Transactions on Machine Learning Research (09/2025)
REAL DATA
LVGM
Views From Real Data
Views From LVGM
(x = cl
f(x1; ()) = e1
(x(x) x
Tal
giz: 0)
(x2; e2
f(x2: -22
{x1
LATENT SPACE
(a)
(b)
MLVGM
Views From MLVGM
Ø) =
23
gZl Z2, Z3, θ)
T (2
(222))
T 2s)
0(x2; -22
MULTIPLE LATENT SPACES
(c)
Figure 3: Superviue Contrastive Representation Learning (SSCRL) optimizes an embedding function f(x Φ) = e, mapping semantically similar images X1, to nearby latent representations 01, C2. (a) In the classic approach, positive views are generated by applying hand-crafted transfornations in the pixel space, Tx, to a finite dataset of images x = [xx] JNN (b) Alternatively, positive views can be generated by sampling nearby points in the latent space of Latent Variable Generative Model (LVGM), g(2; 0). However, since image features remain highly entangled in the latent space, even subtic perturbations (z( may change important characteristics such as dog breed (Australian to Yorkshire Terrier in the figure). (c) Our framework leverages a Multiple Latent Variable Generative Model (MLVGM), represented as g(z,, Z2..... Zn; 0) (n= 3 in the figure). By applying tailored perturbations (2,))) to cach latent variable, we leverage the hicrarchical feature representation to obtain a broader range of valid transformations while maintaining important. semantic aspects.
loading step, which is often the bottleneck in neural network training, as new batches are generated directly into GPU memory; and (iii) maximized diversity, specifically by ensuring that each batch is freshly sampled and never reused, unlike prior methods that rely on fixed-size synthetic dataset.
To evaluate our contributions, we apply our novel quantification algorithm to two distinct MLVGMs: a BigBiGAN (Donahue & Simonyan] 2019) pre-trained on ImageNet- (Dngg et al., 2009),, and StyleGAN2 (Karras et al.j 2020) pre-trained on LSUN Cars (Yu et al. 2015). Subsequently, we leverage the same MLVGMs to generate views for SSCRL using proposed Continuous Sampling (CS) strategy. Specifically, we train feature extractors with three different SSCRL frameworks, namely SimCLR (Chen al. 2020 SimSiam (Chen & Ho. 2021). ; and BYOL (Grill et al. 2020). The learned represcntations are then validated following a consolidated practice through linear classification across multiple downstream datasets and object detection on Pascal VOC (Everingham et al., 2010). Our results demonstrate that MLVGM-based view generation outperforms simple LVGM-basod techniques and achieves comparable or superior results than training with real data. Additionally, we measure the training time per epoch using Contimuuss Sampling us. standard data loading, establishing CS as an efficient alternative for increasing data diversity.
4

---

Published in Transactions on Machine Learning Research (09/2025)
To sum up, our contributions are follows: We propose the first method to quantify the infiuence of individual latent variables in Multiple Latent Variable Generative Models (MIVGMs), which can reveal underutilized latent spaces and serve as a helpful tool for downstream applications. ii) We leverage the natural disentanglement of coarse from fine features in MLVGMs to create positive views for Self-Supervised Contrastive Representation Learning (SSCRL), enabling tailored invariances that outperform previous methods using both real and synthetic data, iii) We introduce Continuous Sampling, a novel procedure that dynamically generates new batches during SSCRL training, increasing data diversity, reducing storage requirements, and maintaining competitive training time performance.
2 Related Works
The idea of utilizing multiple latent variables is well-established in the generative models' literature, typically presented as an evolution of Latent Variable Generative Models (LVGMs). For instance, Variational Autoencoders (VAEs) Kingma & Welling, 2014 Rezende et al.] 2014 leverage multiple latent variables to enhance the expressivity of approximate distributions, as demonstrated by architectures such as NVAE (Vahdat& & Kautz, 2020) and VD-VAE (Child
MLVGMs.
2021 or to improve latent disentanglement, as in Li et al. 2019). Similarly, Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) have embraced this concept in models like LapGAN (Denton et al. 2015). BigGANs (Brock et al. 2018 Donahue & Simonyan
2019). and GigaGAN (Kang et al. (023). Advances in Normalizing Flows (Dinh et al.
2015;
Rezende &
Mohamed 2015) have also incorporated multiple latent variables, with works like Hu et al.
2022
introducing
architectures inspired by physics to achieve this goal.
In such growing environment, numerous applications of MIVGMs have emerged. Of particular relevance in this context is the StyleGAN family (Karras et al.] 2019 2020 2021: Sauer et al. 2022). which has been widely applied in image editing and manipulation tasks (To et al. 2021 Richardson et al. 2021 Alaluf et al. 2022; Pehlivan et al. 2023). More recently, MLVGMs have also been used as foundation models for non non-genenee generatieeeeeee downstream tasks, such as purification against adversarial attacks (Serez et al., 2025 Motivated by such a rapid expansion, in this work we propose a novel application in SSCRL and for the first time study how the "generative load" is spread across latent variables the current architectures.
SSCRL view generation. Self-Supervised Contrastive Representation Learning (SSCRL) (Hadsell et al.) 2006) aims to learn meaningful latent representations without relying on labeled data, primarily by designing informative positive views (Tian et al. 2020 Xiao et al. 2020 Early approaches, such as Bachman et al, (2019); Misra Maaten (2020): Caron et al. (2020), focused on pretext tasks like matching global and local parts of an image create multiple views. Subsequently, SimCLR (Chen et al. 2020 foundational method in the field, introduced the use of manually designed transformations, including flipping, cropping, and color distortions. More recent works have explored advanced techniques, such as learning views in an adversarial manner (Tamkin et al.] 2020 Shi et al.) 2022) or projecting anchor images into the latent spaces
of pre-trained generators (Yang et al. 2022 Astolfi et al. 2023 Kim et al. 2023 Wu et al. 2023 Han et al.) 2023 Zeng et al. 2024). While our approach differs by relying solely on synthetic data, we show that it is also complementary to previous techniques, allowing apply different transformations (denoted as (x) in Figure 3a on top of the generated views-i.e. directly in the pixel space.
Further along our line of work, methods like Jahanian et al. 2021 Li ct al. 2022 have proposed generating fully synthelic views by sampling nearby points in the latent space of pre-trained LVGMs (Figure 3b). We direct compare against these baselines in the experimental section, showing that the coarse-to-fin fine feature disentanglement of MIVGMs allows to obtain better views for most downstream tasks.
Finally, recent efforts have explored the generation of synthelic views in a texl-lo-image setting TTin el al., 2024a b). While this direction holds promise, particularly when combined with MLVGMs, its application our framework remains limited. This is primarily due to the lack of publicly available code and pro-trained models for text-to-image MLVGMs, such as GigaGAN (Kngg et al. 2023
Training with generated data. The remarkable performance of modern generative models, such as Rombach et al. (2022): Chang et al. (0223) has opened up new possibilities for using synthetic data to train
5

---

Published in Transactions on Machine Learning Research (09/2025)
classifier networks. A common strategy involves augmenting real datascts with generated samples. which has shown promise in enhancing classification performance (He et al. 2022 Bansal & Grover 2023 Azizi et al. 2023). Alternatively, more ambitious efforts attempt to train classifiers entirely on synthetic data, leveraging advanced text-to-image models to obtaín high-quality datascts (Sariyildiz et al. 2023 Singh ct al. 2024
The primary challenge in these approaches is the limited diversity of generated data, which has been identified as a key factor contributing the performance gap between classifiers trained on real versus synthetic datasets RRavuri & Vinyals, 2019). Recent studies (Fan et al. 2024 suggest that scaling up the size of synthetic training sets can reduce this accuracy gap, though it does not fully eliminate it. However, generating large datasets introduces its own set of challenges, particularly increased disk space usage and data management overhead. Existing methods (Besnier et al. 2020 Lampis et al.) 2023) address this issue by partially renewing synthetic data at each epoch or by regenerating the dataset entirely every epochs. In this work, we surpass the common assumption about the inefficiencies of generating training images real time, leveraging fast-sampling models, such as GANs, to generate data directly during training and avoiding storage and loading bottlenecks altogether.
3 Methodology
3.1
Measuring the impact of latent variables in MLVGMs
Before formalizing our approach for measuring the contribution of single latent variables, we define the concept of Multiple Latent Variable Generative Models (MLVGMs):
Definition 1 (Multiple Latent Variable Generative Models).
A Multiple Latent Variable Generative Model (MLVGM), denoted g(Z1, Z2...., Zni = x,, is a deep neural network parameterized by 0. It generates new data by modeling random latent variables {21, Z2. 12...... . Zn at different and progressive layers of the network, such that:
g TI. ma x mh x Rm R
g = [[m] (Zn: |n- (zn-1.... ..... (zj(z))...
where 41y represents the ith block of the generalor, and 2i is the corresponding latent variable at that layer (porameters 0 are omilled for clarily).
Intuitions. To meaningfully compare the contribution of each latent variable Zi, we need a metric that operates in the common pixel-space. Probabilistically, we consider images random variable X, and therefore
select Mutual Information (MT) as the metric of choice
As an example, let's consider an MLVGM with n = 3 latent variables, as shown in Figure I Let Z, Zz, Za represent the random latent variables for the three latent spaces, and X the output in the pixel space. Suppose we perturb the first latent variable, replacing Z₁ with Z, This generates modified random variable X' in the pixel space. We now relate the average magnitude of the perturbation in the latent space (e.g. using L2 distance), # E - Z1]22], to the resulting Mutual Information shift in the pixel space, 1(X, = =
The same process can be repeated for Z₂ and Zs, introducing 2. and 2n,, and calculating the perturbation magnitudes #2 and μa needed to achieve the same MI shift y in the pixel space. If the generative process respects the "global-to-loca" hierarchy typically altributed to MLVGMs (Figure Ib), we expect: μa > ui > pl, as depicted in Figure?
Since directly computing MI for high-dimensional variables like X is analytically intractable, we estimate lower bound using InfoNCE (Oord et al.) 2018). Additionally, we employ a Monte Carlo procedure to calculate the average perturbations. Details of these computations are provided in the following sections.
1See Appendix A for the formal definition of Mutual Information and probabilistic interpretation.

---

Published in Transactions on Machine Learning Research (09/2025)
Preliminaries. InfoNCE loss (Ordd et al. 2018 was originally proposed for SSCRL, encouraging similar views (positives) to have close representationss while ensuring that dissimilar views (negatives) remain distant. Formally, it is defined as:
LIsfNNCE Exxx'
log K
(1)
csim(f(xxs)),
where x and x' are the anchor and positive images, respectively, sim denotes the cosine similarity operator, S is the encoder function parameterized by o, is temperature parameter and K is the number of samples (both positive and negative) in a mini-batch.
As demonstrated in Oord et al. (018); Poole al. (2019), InfoNCE provides a lower bound on the MI between the learned representations:
log(2K 1) LInfONCE <IU((x0);
In typical SSCRL setups (e.g, SimCLR (Chen al. 2020 the random variables and X' are generated using deterministic augmentations, such as cropping, flipping, color adjustment, applied to the same base image. These transformations result in fixed mutual information value (X;X') Since ((=;)) is a deterministic function, the fixed term I(X:')) serves as an upper bound to Equation (2) following directly from the data processing incquality (sec Appendix A]:
log(2K 1) LInGNNEE & IIf(X:)) & ((X:').
(3)
Thus, minimizing the InfoNCE loss in SSCRL can be interpreted as tightening the bounds on mutual information, ensuring that the learned representations effectively capture all relevant information shared between the positive views X and X'.
The proposed approach. We build the insights of Equation (3), utilizing InfoNCE as proxy to measure MI shifts between views. Unlike classical SSCRL methods, which rely on fixed, deterministic transformations, we generate views X and X' by perturbing individual latent variables in the latent spaces of a pre-trained MLVGM.
Drawing inspiration from Li al. 2022), which learns latent-space perturbations for positive view generation in LVGMs, our approach adopts an adversarial procedure to optimize InfoNCE loss while progressively reducing the MI between the positive views X and X
Formally, let denote pre-trained MLVGM with n latent variables and parameters 0. The objective is to identify perturbation function (-) for each latent space 1 i n, ensuring that:
(g L2 Zz...., (.. Z2, .2...., (Z.),, Zn; x 7.
(4)
To achieve this, we model (2;);) as simple additive perturbation: (z 2; + p(z: sp), where p(-) is a small multi-layer perceptron (MLP) parameterized by ş. Since InfoNCE provides the lower bound on MI, we need compute it by introducing an encoder function f(-) with paramcters & and define the optimization minimax problem (we omit the parameters 0 of the fixed generator g):
max min LIatONCE (f((())) 22, 24, 2...... (()))) 22,..., Th :)),),
2,,...,2,));0),
(5)
2 detailed discussion of L. al. 2022 provided in Section 3.2

---

Published in Transactions on Machine Learning Research (09/2025)
Training dynamics. We initialize parameters yp such that (-(-) represents the identity function. In other terms, the applied perturbation is initially wero, and the views X and X' are identical. From the perspective of Equation 3 X') = =X),, corresponding to the trivial setting where the encoder f can achieve LInGONCE = O with case. As training continues, the perturbation function (-)-) learns to apply progressively larger modifications to the latent. variable Z5 increasing the diversity of the generated views. This, in turn, reduces the mutual information T(X; X'), thereby lowering the upper bound in Equation (3). As result, the encoder J, tasked with minimizing ListaNCE, must maintain the shared information between increasingly distinct views X and X' into common representation, tighteming the lower bound.
In summary, T progressively enhances diversity in the views, reducing T(X; and causing InfoNCE to increase over time. Conversely, f seeks to learn the most informative representations, tightening the lower bound from the left and seeking equality in Equation (3). We refer the reader to Appendix B for a detailed graphical illustration of these training dynamics, showing the evolution of InfoNCE loss, average perturbations, and additional insights such as required training time and hyperparameter settings.
Monte Carlo sampling. As result of the above, we obtaín independent perturbation functions Ta, (+); acting single latents. For each of these, training is stopped when LIniONCE 7, ensuring views with a consistent MI shift across all latent variables, and allowing the direct comparison of perturbation magnitudes across different latent spaces.
Afterwards, we perform Monte Carlo (MC) sampling by computing a statistically relevant mumber of image pairs for each level i: X = gZ,, Zz,...; -Z,.... Za; 0) and X' = gg,, Zz,..., ......,,7 (Z.),. Zn; O), where we know that [(X; X') 7. This enables estimate the average latent perturbation pr required achieve a similar MI shift in the image space. As depicted in Figure we generally expect that later latent spaces require larger perturbations to achieve the MI shift, matching the empirical observations on the "global-to-local" property of MLVGMs.
In Scction 4 we use this strategy to estimate the impact of latent variables for two distinct MLVGMs: a BigBiGan Donahue & Simonyan (2019) pre-trained on ImageNet-1K Deng et al. (2009) and a StyleGan2 Karras et al. (2020) pre-trained on LSUN Cars Yu et al. (2015). The former has latent variables, while the latter has 16, which we re-organize into 4 groups of 4 for computational practicality.
3.2 Positive view Generation Strategies
As illustrated in Figure 3c, we generate pairs of positive views by applying perturbations to one or more latent spaces, each with an appropriately selected magnitude. To better understand the rationale behind our approach, first look at how previous methods decide view generation strategies.
The problem of Self-Supervised Contrastive Representation Learning (SSCRL) is strictly correlated to designing effective positive views, enabling meaningful representations. In Tian et al. 2220 the following principle is introduced:
Background.
Proposition 1 (Optimal Views for SSCRL, Tian et al (020
Given a dounstream task T with labels Y E V, the optimal views (X;: X²) created from data X are:
X XG) arg min (X1; X2): subject to 1(x1; Y) = 1(2; Y) = ()x;
(6)
X1:X2
meaning that the Mutual Information (MI) between optimal views is minimized to contain only the task-relevant information T(X;: ((;Y)) while removing all nuisance information, I(X;; Y)= 0.
The principle states that optimal views should minimize their Mutual Information (MI) while retaining all
information relevant to the downstream task, expressed by some label Y. However, in SSCRL, labels are unavailable, and the dounstream task is unknown. Consequently, designing optimal views becomes infcasible. To solve this inherent shortcoming, most methods design views by implicitly fixing an MI threshold that decides true positives:
8

---

Published in Transactions on Machine Learning Research (09/2025)
Definition 2 (Mutual Information bound for SSCRL).
Let X1, X2 be two random variables in the common image space. In absence of known downstream task T with labels Y E y; the variables X2 can be considered as positive views if
1(11, 2
(7)
where is an implicitly defined MI threshold.
In other terms, the goal of view generation is to define positives that share a reasonable amount information, relevant for as many tasks as possible. To exemplify this phenomenon, we analyze three prominent positive view generation methods. SimCI.R (Chen al., 2020), defines broad set of data augmentations Tx to be applied in the pixel space. The specific transformations and their combinations are selected through ablation studies conducted the ImageNet- 1K classification task. In this context, the threshold p is implicitly fixed by means of Equation Gb, where a pretext downstream task is considered.
In the context of Latent Variable Generative Models (LVGMs), two methods for generating views by perturbing the single latent space stand out. First, Jahanian et al. (2021) propose applying random perturbations
to an anchor latent variable z. Specifically, the perturbation defined as T2(z) = z + Wrand, where Wrand (N' follows a truncated Gaussian distribution with truncation paramcter t. Similar SimCLR, the parameters of the distribution (e.g., the standard deviation a) are tuned via ablation studies on ImageNet, fixing p with a pretext task.
In contrast, Li et al. 2022) introduce an adversarial approach to learn the perturbation T2 for each instance. In this case, the positive view is generated as (z) = 2 + Wlearn, where Wiearn is a learnable perturbation vector. The objective function is formulated similarly to Equation (5), but applied to LVGMs. In this case, the training stopping criterion (and therefore the threshold p) is empirically decided by observing the quality of generated views at each step.
Deciding perturbation magnitudes. We maintain the perturbation strategies proposed by Jahanian ct al. (2021) and Li et al. (2022). referred to as random and learned, respectively. In fact, the core advantage of our method relies on designing tailored magnitudes based on each latent space's contribution to the generative process, rather than proposing a novel perturbation strategy.
To do so, we proceed with two distinct phases. First, we use the quantification algorithm to relate the impact of different latents. This reveals the presence of over- or under-used codes, which we can immediately discard. Overused codes imply that even very small perturbations will result in low p throshold, while unused codes do not affect the threshold in any way. Therefore, this first step has the goal to discard perturbations that lead to non-informative views. Finally, in the second phase we consider the remaining codes only and visualize the semantic effect of the applied perturbations to estimate valid magnitudes, implicitly defining p.
As an example, in Figure f we generate multiple examples for each considered MLVGM, by perturbing each latent variable (or group, in the case of StyleGan2) with the same latent vector w. For BigBiGan, the quantitative analysis reveals that the first code is overused, and we can verify that even small perturbations completely change the views semantic content. On the opposite, the last latent is underused, and visually presents no differences w.r.t. the reference image. As result, we apply perturbations only on the remaining four latents, which act on small shape details or colors. Similarly, for StyleGan2, we first quantitatively verify that the last group is underused, and discard it. For the remaining ones, we visually observe how the first group controls large-scale transformations, the second group adjusts subject and background composition, while the latter primarily responsible for color variations.
Advantages. The inherent disentanglement of coarse and fine- tne-grained features in MLVGMs offers clear advantage over standard LVGMs, even when employing similar perturbation strategies (random or learned). To clarify, consider the set of all possible downstream tasks for image data, {T1, , }r). For each latent point ~~2 2, there exists a maximum perturbation magnitude |wz]7t for each Lask T,, such that the resulting views are optimal for the task (Proposition
In SSCRL, however, the downstream task Te unknown. Latent perturbation- based positive view generation methods, therefore, aim to define function (z( = -wzl that estimates, for each latent point, non-trivial
9

---

Published in Transactions on Machine Learning Research (09/2023)
anchor
single latent perturbations
anchor
group perturbations
Figure Examples of views generated by adding the same latent vector w to different levels. (a) Two anchor images and possible views generated by the perturbations of 6 BigBiGan's latent levels, represented as the 6 elements' vector the bottom. The darker element indicates the applicd perturbation Ty(,) z; for each latent level i. (b) Generated anchors and views by StyleGan2, which has 16 hierarchical levels, grouped into 4 sets and represented as the 4 elements' vector at the bottom. The darker element indicates the altered group.
perturbation that generates valid (true) positives for as many tasks as possible. In other terms, it exists a direct mapping between the selected latent functions and the choice of the MI threshold pt while larger ww| (lower p) can yield more informative (hard) positives, it also increases the likelihood of producing false posilives, polentially reducing generalization across diverse lasks.
The primary advantage of MLVGMs lies in their hierarchical structure, which distributes features across multiple latent spaces. This enables the definition of separate perturbation functions (.) for each latent space i, tailored to the impact of that space on the generative process. Crucially, the average perturbation magnitude |w|: progressively increases with the latent level index i, as illustrated Figure 3a For example, perturbations in a high-level latent space (responsible for fine details like textures) can be very large without compromising the validity of positive views for most downstream tasks i.e. they have low effect p.
Thus, for an MLVGM with n latent spaces, perturbation magnitudes can be progrossively scaled: |w/1 < |w/2 < .... < (/w. In contrast, LVGMs encode all features within a single latent space, forcing perturbations to be constrained by the most sensitive features. From an MLVGM perspective, this corresponds to a uniform perturbation magnitude, |w| = |w/2 _.... = |W/s which significantly limits flexibility and reduces the impact of generated positive views.
3.3 Continuous Sampling
Utilizing generative models to sample both anchor and positive views can degrade final performance (Ravuri & Vinyals] 2019), primarily due to the lower variability of synthetic images compared to real data. To address this limitation, previous methods (Besnier et al. 2020 Lampis al. 2023 Fan et al. 2024) have proposed increasing variability by sampling larger number of images relative to the reference dataset size, ensuring batches are not repeated across epochs. However, the prevailing approach involves sampling this extensive synthetic dataset offline (before training), which demands significant storage capacity and additional pre-processing time.
In this study, we avoid these drawbacks by adopting a Continuous Sampling strategy that leverages fast generators (such as GANs) to dynamically sample new images during the training of the SSCRL encoder. Specifically, we load the pre-trained generator onto the same GPU device as the encoder and replace the standard data loading step with on-the-fly generator inference step. This process outputs new batch of synthetic images directly on the target device, eliminating the need for pre-generated datasets. Since the pre-trained GAN operates exclusively in inference mode, the additional memory overhead is minimal and affordable, allowing us to maintain sufficiently large batch sizes for elfective SSCRL training.
With this Continuous Sampling approach, the number of training steps per epoch remains consistent with real-data-basee training. However, the total number of unique images seen during training is significantly increased, as the effective training set size becomes n epochs times larger. Moreover, this strategy eliminates
10

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Results the MC simulation on BigBiGan (a) and StyleGan2 (b). For each latent level or group 9 we show the final InfoNCE loss value and the estimated mean (μ) and standard deviation (Ø) of the corresponding inferred distribution. Average perturbation values confirm that early levels greatly impact the generation process, while later levels may have impact at all.
(a)
(b)
latent level
loss
estimated q¹
latent group
loss
estimated g
(8)
(InfoNCE)
(14)
(a.)
(g)
(InfoNCE)
(12)
(Tg)
1
1.09
0.67
0.21
0.99
15.1
2.7
2
1.04
3.63
1.18
1.14
29.0
4.6
3
1.05
6.97
1.85
9 12
0.94
38.0
5.6
4
1.02
13.00
7.08
13
16
0.11
134.4
14.2
5
1.05
21.22
13.68
6
0.14
594.71
616.80
the need for pre-generating and storing extensive datasets and avoids standard data-loading bottlenecks, resulting training times that are comparable or faster than traditional methods (sce Section 4 For a detailed implementation, we provide pseudocode for the continuous sampling procedure in Appendix C
4 Experiments
In this section, present the results of our Monte Carlo procedure for quantifying the impact of latent variables on two MLVGMs: BigBiGan and StyleGan2. Subsequently, we utilize these MLVGMs as view generators to train encoders using different SSCRL frameworks, leveraging our proposed Contimuous Sampling strategy.
To evaluate the effectiveness of our approach, we compare it against two existing latent perturbation techniques for IVGMs, specifically those introduced by Jahanian al. (2021) and Li al. (2022). As additional bascline, include SimCLR, a widely-used view generation method applied to real data, and investigate its combination with trausformations applied on top of MLVGM-generaledd views. Furthermore, in Appendix F we extend the applicability of our method to other generative models beyond GANs by training an NVAE (Vahdat & Kautz, 2020) on the CIFAR-10 10 dataset (Krizhevsky et al. 2009).
Finally, we evaluate the overall training efficiency of Continuous Sampling by comparing runtime per- formance against standard data loading pipelines, demonstrating its capability increase data variability without incurring signifieant computational overhead.
4.1 Impact of Latent Variables
train n separate perturbation functions T (n latent Following the procedure detailed in Section 3.1 levels for BigBiGan and n = 4 latent groups for StyleGan2), optimizing the objective in Equation (5) As visually described in Appendix B the InfoNCE loss rapidly decreases toward zero during the initial training iterations. As the perturbation functions T learn to apply increasing perturbations, the InfoNCE loss rises correspondingly. Training terminated once a value y As achieved, indicating an approximately equal MI shift in the pixel space.
For each latent level group, compute the learned perturbation Wz p(z; sp) across statistically significant number of latent points z. This enables us to estimate the mean (μ; or (1y) and standard deviation (a; or og) of the inferred perturbation distributions q q'((w)) or (²((w)) Table presents these results, along with the final InfoNCE loss achieved during training.
From Table la (Monte Carlo results for BigBiGan), we observe that the average perturbation (estimated mean (4) required to achieve a comparable InfoNCE loss increases progressively across latent levels, from to i 5. However, for = 6, the InfoNCE loss does not rise substantially even under high average
11

---

Published in Transactions on Machine Learning Research (09/2025)
Table Comparison of representations learned on the ImageNet- dataset or BigBiGan generator with two contrastive frameworks (SimCLR and SimSiam). Metrics are Top-1 and Top-5 accuracy for linear classification on ImageNet-IK, average precision for detection on Pascal VOC, and mean Top-1 accuracy over transfer classification datascts. "random" row refers to Jahanian al. (2021). and "learned" to Li et al. (022). Bold indicates the best result for each group, underline the absolute best, and indicates the baseline reported
from Li et al. [2022).
SinCLR
SimSiam
Data Tx
Tx
ImageNet-Il IK
Pascal VOC
ImageNet 1K
Pascal VOC
Transfer
Top-1
Top-5
AP
APs0
AP75
Top-1
Top-5
AP
AP50
APrs
Top-1
real
all all
49.4"
75.6'
52.9"
78.7"
58.5"
49.1
74.2
54.4
80.0
60.0
58.2
synth
41.6*
66.6*
51.0*
77.2*
55.8*
32.2
56.5
51.6
78.2
57.0
47.2
synth random all no col.
48.7"
73.1"
50.2"
77.0"
54.4"
33.4
57.7
51.7
78.4
56.3
47.0
synth ML rand.
53.7
77.2
53.3
79.5
58.5
42.5
67.7
54.3
79.9
59.6
59.6
synth learned all
53.2*
77.2"
53.1*
78.9"
58.0*
33.0
58.2
51.8
78.0
56.7
46.2
synth
ML learn.
col.
54.4
77.9
53.4
79.5
58.9
39.5
64.8
52.5
78.9
57.5
54.9
Table 3: Comparison of representations learned on the LSUN Cars dataset or StyleGan 2 generator with two contrastive frameworks (SimSiam and Byol). Metrics are Top-1 and Top-5 accuracy for linear classification on Stanford Cars and FGVC Aircraft. "random" row refers to Jahanian et al. (202)). and "learned" to
el al. 2022). Bold indicates the best result for each group, underline the absolute best.
SimSiam
Byol
Data
T₂
Tx
Stanford Cars
FGVC Aircraft
Stanford Cars
FGVC Aircraft
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
real
all 33.4
64.3
20.7
48.8
48.9
79.3
35.0
65.6
synth all
27.0
54.6
21.3
50.5
40.5
69.6
31.2
61.9
synth random all 29.2
58.1
22.5
51.7
44.6
73.3
30.5
60.4
syuth
ML rand.
no col. 47.0
76.1
22.9
53.5
58.7
84.8
32.5
61.8
synth
learned all no col.
28.6
56.7
22.0
51.9
45.6
73.6
31.7
62.1
synth
MT. learn.
35.2
64.8
23.0
53.0
47.8
77.1
30.7
61.1
perturbations, suggesting under-utilization of the latent level in the generative process. Conversely, we measure a very low μ4 for the first latent level, suggesting possible over-utilization. These observations may indicate potential inefficiencies in the BigBiGan architecture or training procedure.
A similar trend is observed for StyleGan2 (Table Ib where larger perturbation magnitudes (Pg) are needed achieve comparable InfoNCE loss values as latent groups progress from g to g = 13 16. Notably, the final group exhibits a degenerate behavior, where even large perturbations fail influence the MI of the generated views significantly.
Overall, these results provide clear quantitative evidence that the supposed global-to-local dynamics in MLVGMs hold. Specifically, early latent levels or groups exhibit stronger influence on the generation process, while later ones primarily affect fino-grained details. To the best of our knowlodge, this is the first empirical demonstration such dynamics across MLVGMs.
4.2 View Generation
We test MLVGMs generated views by training multiple ResNet-50 encoders, using SimSiam CChn & He 2021 SimCLR (Chen al.) 2020) (on BigBiGan, following previous work Li al. (2022) and Byol Grill
et al. 2020 (on StyleGan2). We sample latent. anchors from truncated normal distribution: A (0.00, 1.0, 2.0)
12

---

Published in Transactions on Machine Learning Research (09/2023)
offline FFCV
offline
BigBiiGan
Gpu Seconds per Epoch
385
0.161
CO2 Emissions
321.3
C02 Emission Rate
0.135
136.2
257.0
0.108
108.9
192.8 8
0.081
081.7
128.5
064.3
0.027
027.2
000.0
32
0.000
256
000.0
x
128
Batch Size
Batch Size
Batch
Figure 5: Total time (GPU seconds), CO2 emissions rate (grams per second) and total CO2 emissions (grams) for the three tested data loading procedures and different batch sizes.
for BigBiGan and (0.0,, 1.0, for StyleGan2. Positives computed using the random or learned strategies, applied separately on each latent level. Given the MC results reported in Table D we first select the latents to discard. These are the first latent of BigBiGan (overused) and the last latent/grous in both BigBiGan and StyleGan2 (underused). Then, we systematically apply different, perturbation ions to the remaining latents, observing that latents -- in BigBiGan obtain relevant semantic changes with similar perturbation magnitude. For StyleGan2, we apply only tiny perturbations on the first two groups (modify shape and global transformations), and a larger perturbation on the remaining group (alters colors). The specific magniludes, well other hyperparameters,
reported in Appendix D
The representation capabilities of the obtained encoders are compared against several methods: training on synthetic data without latent perturbations T2, the random and learned baselines using single latent spaces, and the upper bound of using real data (1.28M images for ImageNet-1K (Deng et al. 2009) and 893K images for LSUN Cars (Yu et al. 2015)). In all these scenarios, SimCLR pixel-space augmentations Tx are used, consisting of random cropping, horizontal flipping, grayscale, and color jittering. Since our ML views generate realistic color changes (see Appendix HH), we only partially apply Tx transformations top of our positives, removing grayscale and color jittering. To better investigate this aspect, in Appendix P further test various combinations of Tx coupled with our method.
BigBiGan views are evaluated on ImageNet-1k linear classification and, for Simsiam, on seven transfer
datasots: Birdsnap [Berg ct al. 2014), Caltech101 (Fei-Fei et al. 2004), Cifar100 (Krizhessky et al. 2009). DTD (Cimpoi et al. 2014), Flowers102 NNilsback & Zisserman, 2008). Food101 (Bossard el al. 201 4), and
Pets (Parkhi al. (012)). We also compute Average Precision on Pascal VOC (Everingham
al. (0100)
object detection using detectron 2 (Wu et al. 2019)) to train a Faster-RCNN with the R50-C4 backbone. The results are reported in Table including the mean accuracy for the transfer tasks (complete runs in Appendix Gb. For StyleGan2, we compute linear classification accuracy on Stanford Cars (Krause et al. 2013 and FGCV Aircraft 2013b (Maji et al. (013)), reporting results in Table 3
In all experiments, MLVGMs views outperform the corresponding baseline, proving their superior quality. Comparing random and learned methods, we observe that the multiple latent (MI.) random experiments often close the gap with the learned counterparts. This suggests that distinct-level perturbations are more important than the selected alteration technique. In comparison with real data, ML views generally yield better similar results, except in the case of SimSiam encoders evaluated on ImageNet- However, this gap narrows or disappears in other downstream tasks and datasots, evidencing good generalization capabilities of the learned representations, which is the main goal of SSCRL. For StyleGan2, the great performance boost given by ML random views on Stanford Cars noteworthy. When generalizing to FGCV Aircraft, all runs achieve similar performance, with marginal improvements the MI. runs when using SimSiam, and good real data results on Byol. This may be due the high domain shift between the two datasets (Car vs Aircraft), leading to challenging generalization for all representations.
13

---

Published in Transactions on Machine Learning Research (09/2025)
4.3 Continuous Sampling
All our encoders are trained using Continuous Sampling, except for SimCLR, which follows the previous setup. Additionally, to compare overall training speed to standard data loading, we trained ResNet-18 model with SimCLR for 20 epochs on ImageNet-100, on 4 NVIDIA A100-SXM4-40GB GPUs and different batch sizes (32 x 4, 64 x 4, 128 x 4, 256 x 4). The experiment compares the standard PyTorch Paszke et al. (2019) loader, the efficient FFCV Leclere et al. (2023) loader (both with workers), and the BigBiGan generator. Figure D displays our findings, reporting the mcan GPU scconds per epoch, the CO2 emissions rate, and the total CO2 emissions estimated using CodeCarbon Schmidt et al. (2021). Continuous Sampling proves significantly faster than torch loader and only marginally slower than FFCV. In terms of CO2 emissions rate, the use of BigBiGan led to higher energy consumption, due to intensive GPU usage. Nevertheless, in terms of total CO2 the values remain comparable with torch. In conclusion, the increasing efficiency and precision of modern image generation models, especially fast-sampling GANs, make Continuous Sampling an interesting alternative to conventional hata-loading techniques, allowing great image variability while maintaining comparable training times.
5 Discussion and Conclusions
In this paper, we explored the influence of multiple latent spaces in MIVGMs' image generation, quantifying their impact as MI shifts in the common pixel space. This approach advances beyond previous empirical observations, providing deeper insights into the generative process, revealing under- or over-utilized latent variables, and guiding the use of MLVGMs in downstream applications. Additionally, we expanded the use of MLVGMs to new downstream task, which is positive view generation for SSCRL, demonstrating superior results w.r.t. previous methods using single-variable models and competing with real data training. We also introduced Continuous Sampling, which allows using generators as a data source, creating large training sets without requiring significant storage capacity and achieving comparable or faster training times than standard data loading.
Limitations and impact. Our work showcases MLVGMs as a distinct category of models, offering new tools to assess the impact of latent variables. Specifically, the proposed Monte Carlo quantification method supports previous empirical observations on the "global-to-local" nature of MLVGMs, but allows a more in-depth and quantitative analysis. Nonetheless, our algorithm remains limited by the nature of Mutual Information itself, which is invariant under invertible transformations. As a consequence, some small perturbations (e.g. constant scaling to each pixel), will be approximated as 0 shift by our method. However, since the differences in average perturbation between each latent space are usually significant, it is reasonable to assume that such small approximations do not invalidate our findings. More specifically, we reveal that modern based MLVGMs, such as BigGan and StyleGan employ over or under-utilizedd variables in the generative process, setting up a base for possible architectural improvements. In terms of view generation, our method has proven its superiority, surpassing previous perturbation strategies applied to single-variale models. However, it does not address the inherent challenge of SSCRL: views are defined upon "reasonable" thresholds, since "optimal" positives depend on the specific downstream task. Regarding generative models as a data source, they offer potential solutions to issucs associated with real datasots, such as privacy concerns and usage rights Kaissis et al. (2020); DuMont Schütte et al. (2021). However, generative models can inherit biases from the original data Asim et al. (2020), and latent perturbations may amplify them, propagating to downstream models. Therefore, techniques to mitigate these biases could be considered Tan et al. (2020): Teo et al. (2023). As for Continuous Sampling, while it reduces disk usage and data diversity, scaling high image resolutions large models may be very GPU-inlensive and result in high CO2 emissions, limiting usability.
References
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inversion with hypernetworks for real image editing, In IEEE Conference on Computer Vision and Pattern Recognition, pp. 18511-18521, 2022.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Muhammad Asim, Max Daniels, Oscar Leong, Ali Ahmed, and Paul Hand. Invertible generative models for inverse problems: mitigating representation error and dataset bias. In International Conference on Machine Learning, pp. 399 409. PMLR, 2020.
Pictro Astolfi, Arantxa Casanova, Jakob Verbock, Pascal Vincent, Adriana Romero-Soriano, and Michal Drozdzal. Instance-conditioned gan data augmentation for representation learning. ar preprint, 2023.
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in Neural Information Processing Systems, 32:15535 -15545, 2019.
Hritik Bansal and Aditya Grover. Leaving reality to imagination: Robust classification via generated datasets. International Conference on Learning Represenlations, Workshop, 2023.
Thomas Berg, Jiongxin Liu, Scung Woo Lec, Michelle L. Alexander, David W. Jacobs, and Peter N. Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In IEEE Conference on Compuler Vision and Pattern Recognition, 2014.
Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez. This dataset does not exist: training models from generated images. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1-5. IEEE, 2020.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101-mining discriminative components with random forests. In European Conference on Computer Vision, pp. 446-461. Springer, 2014.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin, Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912 9924, 2020.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, José Lezama, Lu Jiang, Ming- Hsugn Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. In International Conference on Machine Learning, pp. 4055 4075, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597 1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. Tn JEEE Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them images. In International Conference on Learning Representations, 2021.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In IEEE Conference on Computer Vision and Pallern Recognilion, pp. 3606 3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Ti-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. Icce, 2009.
Emily Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial networks. Advances in Neural Information Processing Systems, 28:1486-1494, 1494, 2015.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. In International Conference on Learning Representations, Workshop, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Aduances in Newral Information Processing Systems, 32:10542-10552, 2019.
August DuMont Schütte, Jürgen Hetzel, Sergios Gatidis, Tobias Hepp, Benedikt Dietz, Stefan Baner, and Patrick Schwab. Overcoming barriers to data sharing with medical image generation: a comprehensive evaluation. NPJ digital medicine, 4(1):141, 2021.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal on Computer Vision, 88:303 338, 2010.
William Falcon and The PyTorch Lightning team. PyTorch Lightning, March 2019. URL https://githut htwpligghtnnngg-A/iggaaa
Lijie Fan, Kaiseng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. In IEEE Conference on Computer Vision and Pattern 2024.
Recognition, pp.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. IEEE Conference Computer Vision and Paltern Recognition, Workshop, 2004.
lan Goodfellow, Jean Pouget-Abadis Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 27:2672 2680, 2014.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advonces in Neural Information Processing Systems, 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1735 1742, 2006.
Ligong Han, Seungwook Han, Shivehander Sudalairaj, Charlotte Loh, Rumen Dangovski, Fei Deng, Pulkit Agrawal, Dimitris Metaxas, Leonid Karlinsky, Tsui-Wei Weng, al. Constructive assimilation: Boosting contrastive learning performance through view generation strategies. ar preprint, 2023.
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? Tn International Conference Learning Representations, 2022.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017.
Hong-Ye, Ye Hu, Dian Wu, Yi-Zhuang You, Bruno Olshausen, and Yubei Chen. Rg-flow: A hierarchical and explainable flow model based on renormalization group and sparse prior. Machine Learning: Science and Technology, 3(3):035009, 2022.
Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. In International Conference on Learning Representations, 2021.
Georgios A Kaissis, Marcus R Makowski, Daniel Rückert, and Rickmer F Braren. Secure, privacy-preserving and federated machine learning in medical imaging. Nature Machine Intelligence, 2(6):305-33 331, 2020.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-inage synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 10124 10134, 2023.
Tero Karras and Janne Hellsten. stylegan2. http:://ggthuu..cco/////aaaa//// 2019.
Tero Karras and Janne Hellsten. stylegan2-udh--pytorri. https://github.. wwt 2021.
styylegan2-ad--yyee
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. IEEE Conference Computer Vision and Pattern Recognition, pp. 8107-8116, 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellston, Jaakko Lehtinon, and Timo Aila. Alias-Tree generative adversarial networks. Advances Newral Information Processing Systems, 34:852 863, 2021.
Taekyung Kim, Debasmit Das, Seokeon Choi, Minki Jeong, Seunghan Yang, Sungrack Yun, and Changick Kim. Neural transformation network to generate diverse views for contrastive learning. In IEEE Conference on Compuler Vision and Pollern Recognition, pp. 4900 4910, 2023.
Diederík Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
Diederik P Kingma and Max Welling. Autoencoding variational bayes. In International Conference on Learning Representations, 2014.
Jonathan Kranse, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fino-grained categorization. In IEEE Conference on Computer Vision and Pattern Recognition, Workshop, 2013.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. University of Toronlo, 2009.
Andrea Lampis, Eugenio Lomurno, Mattco Matteucci, and Via Ponzio. Bridging the gap: Enhancing the utility of synthetic data via post-processing techniques. In The British Machine Vision Conference, 2023.
Guillaume Leclere, Andrew Tyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. FFCV: Accelerating training by removing data bottlenecks. In JEEE Conference Computer Vision and Pattern Recognition, 2023. https://gi
Yinqi Li, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Optimal positive generation via latent transformation for contrastive learning. Advunces Neural Information Processing Systems, 35: 18327 18342, 2022.
Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali, and Linwei Wang. Progressive learning and disentanglement of hicrarchical representations. In International Conference on Learning Representations, 2019.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint, 2013.
Luke Melas-Kyriazi. pytorch- pretrained-raa... 2021.
Luke Melas-Kyriazi, Christian Rupprocht, Iro Laina, and Andrea Vedaldi. Finding an unsupervisod image segmenter in each of your deep generative models. In International Conference on Learning Representations, 2022.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In IEEE Conference on Computer Vision and Pattern Recognilion, pp. 177776717, 2020.
Maria-Elent Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. Tn Indion Conference on Computer Vision, Graphics & Image Processing, pp. 722-720 IEFE, 2008.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with cont.rastive predictive coding. Xrviv preprint, 2018.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In IEFE Conference on Computer Vision and Pattern Recognition, pp. 3498-3505, 3505. IEEE, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-peeformance deep learning library. In Advonces in Neural Information Processing Systems, pp. 8024-8035. Curran Associates, Inc., 2019.
Hamza Pehlivan, Yusuf Dalva, and Aysegul Dundar. Styleres: Transforming the residuals for real image oditing with stylegan. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1828-1837, 2023.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 1171-5180. PMTR, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Representations, 2016.
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. Advances in Neural Information Processing Systems, 00112268-12270 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, pp. 1530-1538. PMLR, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, pp. 1278-1286. PMLR, 2014.
Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski. Kornia: an open source differentiable computer vision library for pytorch. In Proceedings of the IEEE/CVE Winler Conference on Applications of Computer Vision, 2020.
Flad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-ing to-iagee translation. In IEEE Conference on Computer Vision and Pollern Recognition, pp. 2287 2296, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-11, 2023.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. ACM SIGGRAPH Conference Proceedings, 2022.
Victor Schmidt, Kamal Goyal, Aditya Joshi, Boris Feld, Liam Conell, Nikolas Laskaris, Doug Blank, Jonathan Wilson, Sorelle Friedler, and Sasha Luccioni. Codecarbon: estimate and track carbon emissions from machine learning computing (2021). DOT: htlps://doi. 11110.52112eene 4658424, 2021.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Dario Serez, Marco Cristani, Alessio Del Bue, Vittorio Murino, and Pictro Morerio. Pre-trained multiple latent. variable generative models are good defenders against adversarial attacks. In Proceedings of the LEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6506 6516. IEEE, 2025.
Yuge Shi, N Siddharth, Philip Torr, and Adam R Kosiorek. Adversarial masking for self-supervised learning. In International Conference on Machine Learning, pp. 20026-20040. PMLR, 2022.
Krishnakant Singh, Thanush Navaratnam, Jannik Holmer, Simone Schaub-Meyen and Stefan Roth. Is synthetic data all we need? benchmarking the robustness of models trained with synthetic images. In 2024.
IEEE Conference on Computer Vision and Paltern Recognilion, pp-
Alex Tamkin, Mike Wu, and Noah Goodman. Viewmaker networks: Learning views for unsupervised representation learning. In International Conference on Learning Representations, 2020.
Shuhan Tan, Yujum Shen, and Bolei Zhou. Improving the fairness of deep generative models without retraining. arXiv preprint, 2020.
Christopher TH Teo, Milad Abdollahzadeh, and Ngai-Man Cheung. Fair generative models via transfer learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 2429-2437, 2023.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for coutrastive learning? Advances in Neural Information Processing Systems, 33:6827 6839, 2020.
Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, and Phillip Isola. Learning vision from models rivals learning vision from data. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 15887 15898, 2024a.
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems, 36, 2024b.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashník, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions Graphics, 40(4):1-14, 2021.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in Newral Information Processing Systems, 33:19667-19673 2020.
Yawen Wu, Zhepeng Wang, Dewen Zeng, Yiyu Shi, and Jingtong Hu. Synthetic data can also teach: Syuthesizing effective data for unsupervised visual representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 2874-2874, 2023.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. https 2019.
Tete Xiao, Xiaolong Wang, Alexci A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In International Conference on Learning Representations, 2020.
Yu Yang, Wing Yin Cheung, Chang Liu, and Xiangyang Ji. Local manifold augmentation for multiview semantic consistency. arXiv preprint, 2022.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction large-scale image dataset using deep learning with humans in the loop. arXie preprint, 2015.
Dewen Zeng, Yawen WI, Xinrong Hu, Xiaowei Xu, and Yiyu Shi. Contrastive learning with synthetic positives. In European Conference on Computer Vision, pp. 430-447, 2024.
19


Reviews:
[{"review_id": "KAm39WYqBI", "reviewer": "Reviewer_jyeL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nI found the paper to be very well-written and easy to read. The authors did a good job at sharing useful insights regarding the impact of latent perturbations in MLVGMs. I appreciate the ablations on perturbations strengths and MI thresholds. The approach to generative views is very effective and the downstream results with contrastive learning indicate that.\n\n**Weaknesses**\n\nWhile the paper demonstrates good results for positive views generation via their perturbations of MLVGMs, I have a few concerns.\n\n- The authors do not study an explicit mapping from norm size to types of semantic change in pixel space. There is thus interpretability gap for perturbation i.e., the optimisation treats the perturbation as “smallest L2-norm offset that lowers MI,” but because the direction is unconstrained, the same-sized perturbation may manifest as colour shifts, pose changes, or texture noise; hence magnitude alone does not map to a specific, repeatable semantic transformation, which limits the method’s utility for tasks that demand controlled edits.\n\n- There has been some work on identifying latent directions and consequently leveraging them for interpretable, controlled generation in GAN which is very relevant to this work [1, 2]. It would be interesting to compare these methods for the same SSCRL tasks?\n\n\n- Continuous Sampling is claimed to circumvent storage and sample diversity issue. While authors show that online generation in continuous sampling is on par or better than dataloading. It adds other issues for e.g., \n1. is final accuracy of a deterministic contrastive downstream model reproducible given on-the-fly stochasticity in training?\n2. The latency also depends on the generative model and with bigger model it would be hard to scale it.\n3. The generative model will occupy precious gpu compute that can be in turn used for increased batch size for contrastive model training. Have authors looked into this aspect?\n\n\nReferences:\n1. GANSpace: Discovering Interpretable GAN Controls [Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, Sylvain Paris]\n2. Optimal Positive Generation via Latent Transformation for Contrastive Learning [Yinqi Li, Hong Chang, Bingpeng MA, Shiguang Shan, Xilin Chen]", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a systematic procedure to quantify the impact of individual latent variables within Multiple Latent Variable Generative Models (MLVGMs) while using InfoNCE (at embeddings level) as a proxy to measure MI shifts between images. Building on this, while leveraging hierarchical disentanglement within MLVGMs, the authors introduce an approach to generate positive views for Self-Supervised Contrastive Representation Learning (SSCRL). They further introduce a continuous sampling strategy which dynamically generates synthetic data during training, aiming to enhance data diversity. Experiments on BigBiGAN, StyleGAN-2 coupled with SimCLR/SimSiam/BYOL demonstrate the method's effectiveness, often outperforming the contemporary view-generation strategies such as image augmentations and using pre-generated synthetic-views.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have following questions. They are not typical \"requested changes\" but any follow-up explanation/experiment would be helpful.\n\n- Could the latent-space perturbation functions be constrained to specific Lie-group actions (e.g., translation, rotation, isotropic scaling) so that each learned direction corresponds to a single geometric transform on the image? If so, how would you enforce and verify that disentanglement?\n\n- Positive-pair definitions can be application-dependent (e.g., background variance with foreground fixed). Can your framework target such feature-conditioned perturbations—changing class-irrelevant factors while preserving class-critical ones—and how do you label or discover which latent offsets achieve each effect?\n\n- Beyond positives, could the same MI-based metric be inverted to sample hard negatives—perturbations that leave low-level appearance similar yet flip high-level semantics?\n\n- The paper assumes a direct relationship between the L2 norm of a latent perturbations and the resulting MI shift in the pixel space. Can’t the perturbations not introduce changes that are can/will not be explained by the mutual information in pixel space? for e.g., how would account for latent perturbation that change illumination or texture in ways MI fails to capture?\n\n- Given the probabilistic nature of the generator, do identical latent perturbation sometimes yield different MI shifts across stochastic draws? If so, how is this variability reflected in your ranking of “important” vs. “under-utilised” latents?\n\n- When InfoNCE embeddings are already trained for invariance, does measuring MI in that embedding space risk underestimating meaningful pixel-space changes? Why not evaluate MI directly between image distributions (e.g., with MINE) to avoid embedding-induced blind spots?", "broader_impact_concerns": "- While continuous sampling reduces disk IO, continuous sampling of high-resolution images from large generators is still GPU-intensive. The authors do study CO2 emission explicitly but I am afraid that any approach to scale up this method would be detrimental to environment.\n\n- Synthetic-data biases – Latent perturbations may amplify biases present in the generator (e.g. over-representation of certain demographics) and downstream models might inherit those biases unnoticed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yk6NPxeghv", "reviewer": "Reviewer_Smc8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The paper presents a novel and principled approach to quantifying the flow of MI across hierarchical latent spaces in MLVGMs.\n- The adversarial perturbation method provides a structured way to analyze information bottlenecking at different levels, offering a deeper understanding of global-to-local MI compression.\n- The proposed framework is well-motivated and theoretically sound, with a clear focus on quantifying MI variations in hierarchical generative models.\n- The paper extends its analysis to contrastive learning, proposing a new approach for generating positive views to improve representation learning.\n- The experimental evaluation demonstrates the feasibility of the method, primarily on GAN-based architectures.\n\n### Weaknesses\n- The connection between the MI analysis and its application to contrastive learning is unclear. The contrastive learning contribution appears disconnected from the MI perturbation framework, and it is not immediately evident how the first method informs the second.\n- The writing quality and organization need significant improvement. The paper is highly redundant, repeatedly cycling through the same claims, which disrupts readability and makes it difficult to pinpoint the main contributions within each section. A major revision to improve clarity and flow and delete redundant text is recommended.\n- The paper presents MLVGMs as a general class of models encompassing GANs and VAEs, yet the empirical evaluation is conducted only on GAN-based architectures. This limits the scope of the claims regarding the applicability of the proposed method to MLVGMs in general.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores how mutual information is distributed across latent spaces in multiple latent variable generative models (MLVGMs). The authors propose a principled approach to quantify the “global-to-local” information bottleneck by measuring mutual information (MI) after perturbations at different levels. To achieve this, they introduce an adversarial method that learns perturbations at each hierarchical level, ensuring a constant deviation in MI across levels. Additionally, the paper extends this analysis to the domain of contrastive learning, where the authors propose using their framework to generate positive views for contrastive representation learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Based on the aforementioned weaknesses, I suggest the following revisions, ordered by priority:\n- Improve clarity, structure, and flow. Each section should focus on addressing a specific aspect of the paper. Identifying and removing redundant text would enhance readability and make the contributions clearer.\n- Clarify the connection between the information-theoretic framework for learning balanced perturbations and the view generation approach for contrastive learning. This relationship should be presented in a more structured and principled manner to improve coherence.\n- Evaluate the method on Hierarchical VAEs, such as NVAE or VDVAE (missing reference). This is particularly relevant since the proposed method could benefit from fine-tuning a pre-trained encoder in these architectures.\n\nReference:\n\n[1] Child, Rewon. “Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images.” International Conference on Learning Representations (ICLR).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "naFlCEmkrr", "reviewer": "Reviewer_KaX2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I had to read almost the entire paper to understand what it was really about. And I'm still not sure I entirely understand what exactly the first experiment about mutual information proves or enables. The abstract and introduction could do a better job of clearly explaining what the authors did, why they did it, and what conclusions they drew.\n\nTheir first method and experiment, about training latent perturbation models to achieve similar contrastive losses across layers, feels very complicated compared to the insights it cashes out in. AFAICT, the takeaway from these experiments is effectively that you need to apply much larger perturbations to the later latent variables than the earlier ones in order to achieve the same level of deviation from the original image. It feels like qualitatively similar conclusions could have been reached using far simpler experiments. For example, you could just plot the LPIPS between the original and perturbed images as a function of the magnitude of a random perturbation to each latent. I feel like this very simple experiment would probably have yielded the same final insight as this complicated adversarial contrastive learning training procedure.\n\nA few times in the abstract and introduction, the authors imply that the results of their first experiment about mutual information informed the way they trained their SSRCL in the second experiment. I am not convinced that the mutual information-based analysis of MLVGMs is actually informing your SSCRL method for generating positive views. Indeed the conclusion from your discussion in section 3.2 appears to be that this analysis is ultimately inadequate for determining what perturbations to use. So what was the utility of that mutual information analysis described in section 3.1? It seems like the final takeaway was mainly how much to perturb each latent, but it's not clear precisely *how* the results from table 1 informed those hyperparameter choices from appendix D.\n\nTables 2 and 3 do convincingly demonstrate that the author's MLVGM-based positive view generation usually outperforms their baseline based on more standard augmentations.\n\nNot convinced that Continuous Sampling counts as a technically novel contribution to the field. Feels more like an implementation detail of their training setup.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1) the authors introduce a contrastive loss based method for quantitatively evaluating the impact of the different latent varibale perturbations in Multi Latent Cariable Generative Models MLVGMs. \n2) they use MLV-GMs to generate positive samples for self-supervised contrastive representation learning.\n3) they generate these training samples in real time instead of saving them to disk before training their Self-Supervised Contrastive Representation Learning SSCRL models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "All of these suggestions are made with low confidence; follow them if they seem right to you:\n\nMLVGM and SSCRL are some really rough acronyms. My brain regards them like chaotic piles of capital letters. Maybe a hyphen would help, MLV-GMs? \n\nDefinition 1: Not sure I agree with this use of the word \"random\", IIUC, at least for stylegan, the latent z's {z1,...zn} are themselves pulled from a learned joint distribution and probably have high mutual information. Is it fair to call them \"random\"?\n\nAn odd detail about their setup: if T(z) is a deterministic function, and assuming g(z1,...zn) is invertible, then the true ideal InfoNCE loss between I1 = f(g(z1,...zi,...zn)) and I2 = f(g(z1,...T(zi),...zn)) should always be... zero? Since there exists a deterministic function from I1 to I2. Any loss greater than zero just means that your encoder f has failed to learn this function. Would be more principled if T(z) included some random noise as input?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)