Unsupervised Learning of Structured Representations via Closed-Loop Transcription
Authors: Shengbang Tong
Xili Dai
Yubei Chen
Mingyang Li
ZENGYI LI
Brent Yi
Yann LeCun
Yi Ma
Venue: Submitted to ICLR 2023 (2023)
Keywords: Unsupervised Conditional Image Generation
Self-supervised Generative Model
Closed-Loop Transcription

Abstract: This paper proposes an unsupervised method for learning a unified representation that serves both discriminative and generative purposes. While most existing unsupervised learning approaches focus on a representation for only one of these two goals, we show that a unified representation can enjoy the mutual benefits of having both. Such a representation is attainable by generalizing the recently proposed closed-loop transcription framework, known as CTRL, to the unsupervised setting. This entails solving a constrained maximin game over a rate reduction objective that expands features of all samples while compressing features of augmentations of each sample. Through this process, we see discriminative low-dimensional structures emerge in the resulting representations.  Under comparable experimental conditions and network complexities, we demonstrate that these structured representations enable classification performance close to state-of-the-art unsupervised discriminative representations, and conditionally generated image quality significantly higher than that of state-of-the-art unsupervised generative models.

Under review as a conference paper at ICLR 2023
UNSUPERVISED LEARNING OF STRUCTURED REPRE- SENTATIONS VIA CLOSED-LOOP TRANSCRIPTION
Anonymous authors Paper under double-blind review
ABSTRACT
This paper proposes unsupervised method for learning unified representation that serves both discriminative and generative purposes. While most existing unsu- pervised learning approaches focus on representation for only one of these two goals, we show that a unified representation can enjoy the mutual benefits of having both. Such representation is attainable by generalizing the recently proposed closed-loop transcription framework, known as CTRL, to the unsupervised setting. This entails solving a constrained maximin game over rate reduction objective that expands features of all samples while compressing features of augmentations each sample. Through this process, we see discriminative low-dimensional structures emerge in the resulting representations. Under comparable experimen- tal conditions and network complexities, we demonstrate that these structured representations enable classification performance close state-of-the-er art unsuper- vised discriminative representations, and conditionally generated image quality significantly higher than that of state-of-the-ar unsupervised generative models.
INTRODUCTION
In the past decade, we have witnessed an explosive development in the practice of machine learning, particularly with deep learning methods. A key driver of success practical applications has been marvelous engineering endeavors, often focused on fitting increasingly large deep networks to input data paired with task-specific sets of labels. Brute-force approaches of this nature, however, exert tremendous demands on hand-labeled data for supervision and computational resources for training and inference. As result, an increasing amount of attention has been directed toward using self- supervised or unsupervised techniques to learn representations that can not only learn without human annotation effort, but also be shared across downstream tasks.
Discriminative versus Generative. Tasks in unsupervised learning typically separated into two categories. Discriminative ones frame high-dimensional observations as inputs, from which low- dimensional class latent information be extracted, while generative ones frame observations as generated outputs, which should often be sampled given some semantically meaningful conditioning.
Unsupervised learning approaches targeted at discriminative tasks mainly based on key idea: to pull different views from the same instance closer while enforcing a non-collapsed representation by either contrastive learning techniques (Chen et al., 2020b He et al.) 2020: Grill et al., (020a), covariance regularization methods (Bardes et al. 2021 Zbontar et al. [202), or using architecture design (Chen & He 2020 Grill al. (020bb. Their success typically measured by the accuracy of simple classifier (say shallow network) trained on the representations that they produce, which have progressively improved over the years. Representations learned from these approaches, however, do not emphasize much about the intrinsic structure of the data distribution, and have not demonstrated success for generative purposes.
In parallel, generative methods like GANs (Goodfellow et al. 2014) and VAEs (Kingma & Welling 2013) have also been explored for unsupervised learning. Although generative methods have made striking progress in the quality of the sampled or autoencoded data, when compared to the aforemen- tioned discriminative methods, representations learned with these approaches demonstrate inferior performance in classification.
Toward Unified Representation? The disparity between discriminative and generative approaches in unsupervised learning, contrasted against the fundamental goal of learning representations that are useful across many tasks, leads to a natural question that we investigate in this paper: in the unsupervised setting, is it possible learn unified representation that is effective for both

---

Under review as a conference paper at ICLR 2023
discriminative and generative purposes? Further, do they mutually benefit each other? Concretely, we aim to learn a structured representation with the following two properties:
1. The learned representation should be discriminative, such that simple classifiers applied to learned features yield high classification accuracy.
2. The learned representation should be generative, with enough diversity to recover raw inputs, and structure that can be exploited for sampling and generating new images.
The fact that human visual memory serves both discriminative tasks (for example, detection and recogmition) and generative or predictive tasks (for example, via replay) (Keller & Mrsic-Flogel 2018, Josselyn & Tonegawa 2020 Ven et al.] (020) indicates that this goal is achievable. Beyond being possible, these properties are also highly practical successfully completing generative tasks like unsupervised conditional image generation (Hwang et al. [002), for example, inherently requires that learned features for different classes be both structured for sampling and discriminative for conditioning. On the other hand, the generative property can serve as a natural regularization avoid representation collapse.
Closed-Loop Transcription via a Constrained Maximin Game. The class of linear discriminative representations (LDRs) has recently been proposed for learning diverse and discriminative features for multi-class (visual) data, via optimization of the rate reduction objective (Chan et al., 2022). the supervised setting, these representations have been shown to be be both discriminative and generative learned in a closed-loop transcription framework via a maximin game over the rate reduction utility between an encoder and decoder (Dai et al. 2022). Beyond the standard joint learning setting, where all classes are sampled uniformly throughout training, the closed-loop framework has also been successfully adapted to the incremental setting (Tong et al. (022), where the optimal multi-class LDR is learned one class at time. In the incremental (supervised) learning setting, one solves a constrained maximin problem over the rate reduction utility which keeps learned memory of old tasks intact (as constraints) while learning new tasks. It has been shown that this new framework can effectively alleviate the catastrophic forgetting suffered by most supervised learning methods.
Contributions. In this work, we show that the closed-loop transcription framework proposed for learning LDRs in the supervised setting (Chan et al., 2022) can be adapted to a purely unsupervised setting. In the unsupervised setting, we only have to view each sample and its augmentations as "new class" while using the rate reduction objective to ensure that learned features are both invariant to augmentation and self-consistent in generation; this leads to a constrained maximin game that similar to the one explored for incremental learning (Tong et al., 2022). Our overall approach is illustrated in Figure
As we experimentally demonstrate in Section 4 our formulation benefits from the mutual benefits of both discriminative and generative properties. It largely bridges the gap between two formerly distinct set of methods: by standard metrics and under comparable experimental conditions, it enables classification performance close to discriminative methods and unsupervised conditional generative quality significantly higher than state-of tfate-of-the-ar techniques. Coupled with evidence from prior work, this suggests that the closed-loop transcription through the (constrained) maximin game between the encoder and decoder has the potential to offer a unifying framework for both discriminative and generative representation learning, across supervised, incremental, and unsupervised settings.
Method
Lincar Probe
Image Generation
UCIG
SimCLR (Chen et
2020b
X
MOCO-V2 (He ct al.
2020
x
ContraD (Jonn & Shin
2021
x
PATCH-VAS VAE (Parmar ct al. 2021
x
CTRL-Binary (Dai et al. 2022
x
SLOGAN (Hwang et al.
2021
U-CTRL (ours)
Table 1: Comparison of the downstream task capabilities of different unsupervised learning methods. UCIG refers to Unsupervised Conditional Image Generation (Hwang et al. 2021).
2

---

Under review as a conference paper at ICLR 2023
High-dim data space
Low-dim LDR feature space
envoder
7(2²)
R
X
f(2,0)
Z
X
2
closabloop
tanscrigtion
RDD
g(=, 1))
decoder
Figure I: Overall framework of closed-loop transcription for unsupervised learning. Two additional constraints are imposed on the Binary-CTRL method proposed in prior work (Dai et al. 2022): 1) self-consistency for sample-wise features =' and # say z² 2 and 2) invariance/similrity among features of augmented samples z² and x say z x z = f(T((')) 0), where x', = r(x') (zx) is an augmentation of sample x' via some transformation T(-).
2 RELATED WORK
Our work mostly related three categories of unsupervised learning methods: (1) sulf-supe learning via discriminative models. (2) self-supervised learning via generative models, and (3)
compares the capabilities of models learned by various representative unsupervised learning methods.
unsupervised conditional image generation.
SuperSuue Learning for Discriminative Models. On the discriminative side, works like SimCLR (Chen et al.) 2020b), MoCo (He et al.] 2202)) and BYOL (Grill et al. 2020a) have recently shown overwhelming effectiveness in learning discriminative representations of data. MoCo (He) et al. 2020) and SimCLR (Chen et al. (2200b) seek to learn features by pulling together features of augmented versions of the same sample while pushing apart features of all other samples, while BYOL (Grill et al. 2020a) trains student network to predict the representation of teacher network
in a contrastive setting. Barlow (Zbontar et al. 2021) and TCR (Li et al. 2022) learn by regularizing the covariance matrix of the embedding. However, features learned by this class of methods are typically highly compressed, and not designed to be used for generative purposes.
Supervied L Learning with Generative Models. On the generative side, the original GAN GGodffelow et al., 2014) can be viewed as a natural self-supervised learning task. With an additional linear probe, works like DCGAN (Radford et al. 20155 have shown that features in the discriminator can be used for discriminative tasks. To further enhance the features, extensions like BiGAN (Donahue et al., 2016) and ALI (Dumoulin et al. 2016) introduce a third network into the GAN framework, aimed at learning an inverse mapping for the generator, which when coupled with labeled images can be used to study and supervise semantics in learned representations. Other works like SSGAN (Chen et al. 2019), SSGAN-LA (Hou al.) 2021). and ContraD (Jeong & Shin 202T) propose to put augmentation tasks into GAN training to facilitate representation learning. Outside of GANs, variational autoencoders (VAEs) have been adapted to generate more semantically meaningful representations by trading off latent channel capacity and independence constraints with reconstruction accuracy (Higgins al. 2016), idea that has also been incorporated into recognition improvements using patch-level bottlenecks (Gupta et al. [2000), which encourage a VAE to focus on useful patterns in images. By incorporating data-augmentation, VAE is also shown to achieve fair discriminative performance (Falcon et al. 20221) Recently, works like MAE (He et al. 202T) and CAE (Chen al. 2022) have learned representations by solving masked reconstruction tasks using vision transformers. Autogressive approaches like iGPT (Chen et al.) 2020a) have also demonstrated decent self-supervised learning performance, which improves further with the incorporation of contrastive learning (Kim et al. [222). However, unless supervised, features learned by those previously mentioned methods either do not have strong discriminative performance, or cannot be directly exploited to condition the generative task.
Unsupervised Conditional Image Generation (UCIG). For generative models, we often want to be able to generate images conditioned on certain class or style, even in a completely unsupervised setting. This requires that the learned representations have structures that correspond to the desired conditioning. InfoGAN (Chen et al.) 2016)) proposes to learn interpretable representations by maxi- mizing the mutual information between the observation and a subset of the latent code. ClusterGAN (Mukherjee al. 2019) assumes discrete Gaussian prior where discrete variables defined as
3

---

Under review as a conference paper at ICLR 2023
one-hot vector and continuous variables are sampled from Gaussian distribution. Self-Conditioned
GAN (Liu et al. 2020) uses clustering of discriminative features as labels to train. SLOGAN (Hwang
et al. 2021 proposes
conditional contrastive loss (U2C) to learn latent distribution of the data.
Note that compared to our work, ClusterGAN and SLOGAN introduce an additional encoder that leads to increased computational complexity. On the VAE side, works like VaDE (Jian al. 2016) cluster based on the learned feature of a supervised ResNet. Variational Cluster (Prasad et al. 2020 simultaneously learns prior that captures the latent distribution of the images and posterior to help discriminate between data points in end-to-ent unsupervised setting. In this work, we will see how clusters can be estimated principled way more unified framework, by optimizing the same type of objective function that we use for learning features.
3 METHOD
3.1 PRELIMINARIES; RATE REDUCTION AND CLOSED-LOOP TRANSCRIPTION
Assumptions on Data. Our work, as well as prior work in closed-loop transcription (Dai et al.
xx RDD sampled from classes. Borrowing notation from Yu et al. 2020). the membership of the N samples in the k classes denoted using & diagonal matrices: II I E R NxNN 15=1. where the diagonal entry II, (i, of II the probability of sample i belonging to subset j. Let = (II I II 2 0.) be the set of all such matrices. WLOG, we may assume that classes are separable, with images for each belonging to a low-dimensional submanifold in the space RDD
2022 Tong et al., 2022), considers a of N images X ==xx" , x²2
Unsupervised Discriminative Autoencoding. The goal of transcription is to learn unified rep- resentation, with the structure required to both classify and generate images from these i classes. Concretely, this is achieved by learning two continuous mappings: (1) encoder parametrized by 6:
_(, 0) + Rd with d < D such that all samples are mapped to their features as X f( Z with Z = 12
C Rd. and (2) an inverse map o(.,n) x + I E R² such that and á g(f())) is close. In other words, X f(a.0) Z a(z,9) X forms autoencoding.
In this work, we specifically learn this mapping in an entirely unsupervised fashion, without knowing the ground-truth class labels II at all. As stated in the introduction, a both discriminative and generative representation is difficult to achieve by standard generative methods like VAEs and GANs. This is one of the motivations for the closed-loop transcription framework (CTRL) proposed by (Dai et al. 2022). which we will generalize to the unsupervised setting.
Maximizing Rate Reduction. The CTRL framework (Dai et al. 2022 was proposed for the supervised setting, where it aims to map each class onto an independent linear subspace. As shown in (Yu et al. 2020), such a linear discriminative representation (LDR) can be achieved by maximizing coding rate reduction objective, known as the MCR² principle:
1
d
d
AR Z I
2 log det I
ZZ Ne
log det
I+ tr(II,)
ZI1 Z
(1)
2N
R(Z)
R=
where each II, encodes the membership of the N samples described before. As discussed in (Chan) et al. 2022), the first term () measures the total rate (volume) of all features whereas the second term 2t measures the average rate (volume) of the k components. Our work adapts this formula to design meaningful objectives in the unsupervised setting.
Closed-Loop Transcription. To learn the autoencoding X f(a.0), Z g(z.n) X. fundamental
question is how measure the difference between X and the regenerated X g( f((f())) It is typically very difficult to put a proper distance measure the image space (Wang et al.) 2004). To bypass this difficulty, the closed-loop transcription framework (Dai et al. 2022) proposes to measure the difference between X and X through the difference between their features Z and 2 mapped through the same encoder:
f(x,0) Z
9(x.n)
f(x 2.
X
(2)
The difference can be measured by the rate reduction between Z and 2. a special case of [I] with
k 2 classes:
4

---

Under review as a conference paper at ICLR 2023
AR(Z, 2) = -
R()).
(3)
Such AR is a principled distance between subspace-liks like Gaussian ensembles, with the property that A AR(2 ( 2) 0 iff Cov Cov (2) Ma al. 2007).
As shown in (Dai et al. 2022). applying this measure in the closed-loop CTRL formulation can already learn decent autoencoding, even without class information. This is known as the CTRL-Binary program:
max min
AR(Z, 2)
(4)
However, note that 4 is practically limited because only aligns the dataset X and the regenerated X at the distribution level. There is no guarantee that for each sample z would be close to the decoded a = (((())), For example, (Dai al. 2022) shows that a car sample can be decoded into horse; the so obtained (autoencoding) representations are not sample-wise selffconsistem! 3.2 SAMPLE- WISE CONSTRAINTS FOR UNSUPERVISED TRANSCRIPTION
To improve discriminative and generative properties of representations learned in the unsupervised setting, we propose two additional mechanisms for the above CTRL-Binary maximin game { For simplicity and uniformity, here these will be formulated as equality constraints over rate reduction measures, but in practice they can be enforced softly during optimization.
Sample-wise Self- Ceff-Consistency via Closed-Loop Transcription. First, to address the issue that CTRL-Binary does not learn a sample-wise consistent autoencoding, we need to promote ø to be close to a for each sample. In the CTRL. framework, this can be achieved by enforcing that their corresponding features = f fxx and = ((t) are the same or close. To promote sample-wise self-consistency, where E = g f (x)) is close to æ want the distance between z and & to be zero or small, for all samples. This can be formulated using rate reduction; note that this again avoids measuring differences in the image space:
EAREE, O.
Self-Supervision via Compressing Augmented Samples. Since we do not know any class label information between samples in the unsupervised settings, the best we can do is view every sample and its augmentations (say via translation, rotation, occlusion etc) as one "class" - a basic idea behind almost all supervupe learning methods. In the rate reduction framework, it is natural to compress the features of each sample and its augmentations. In this work, we adopt the standard transfornations in SimCLR (Chen et al. 2020b) and denote such a transformation as т. We denote each augmented sample aa T(x). and its corresponding feature as 2,, f(x... 0). For discriminative purposes, we hope the classifier invariant to such transformations. Hence is natural to enforce that the features z. of all augmentations are the same as that z of the original sample æ. This is equivalent to requiring the distance between z and Za, measured in terms of rate reduction again, to be zero (or small) for all N samples:
Σ AR(Z', 2 0.
JEN
3.3 UNSUPERVISED REPRESENTATION LEARNING VIA CLOSED-LOOP TRANSCRIPTION
So far, we know the CTRL-Binary objective AR(Z, 2) in 4 helps align the distributions while sample-wise self-consistency (5) and sample-wise augmentation associated with each sample. Besides consistency, we also want learned representations discriminative for different samples (here viewed as different "classes"). Notice that the rate distortion term R( Z) measures the coding rate (hence volume) of all features. It has been observed in (Li et al. 2022b that by maximizing this term, learned features expand and hence become more discriminative.
help align and compress features maximally
Unsupervised CTRL. Putting these elements together, we propose to learn a representation via the following constrained maximin program, which we refer to as unsupervised CTRL (U-CTRL)):
max min
R(Z)= + R Z, 2)
9 ?
subject to
Σ AR 2¹) = 0, and Σ AR(2', 2-) = 00 ieN
EN
5

---

Under review as a conference paper at ICLR 2023
In practice, the above program can be optimized by alternating maximization and minimization between the encoder f(., 8) and the decoder g(: .7.. We adopt the following optimization strategy that works well in practice, which is used for all subsequent experiments real image datasets:
max R(Z) + AR Z 2)
A
Σ ENN
AR(Z 2
X2
Σ
AR(z', 21;;
(8)
8
min R(Z) + AR Z + A
AR
A2 AR 29).
(9)
IEN
iEN
where the constraints EiEN A AR(2*, 20) 0 and Σ EEN AR 2 = 0 in have been converted (and relaxed) to Lagrangian terms with corresponding coefficients \ and X2
Unsupervised Conditional Image Generation via Rate Reduction. The above representation is learned without class information. In order to facilitate discriminative or generative tasks, it must be highly structured. As we will see via experiments, specific and unique structure indeed emerges naturally in the representations learned using U-CTRL: globally, features of images in the same class tend to be clustered well together and separated from other classes (Figure 2) locally, features around individual samples exhibit approximately piecewise linear low-dimensional structures (Figure (5)
The highly-structured feature distribution also suggests that the learned representation can be very useful for generative purposes. For example, we can organize the sample features into meaningful clusters, and model them with low-dimensional (Gaussian) distributions or subspaces. By sampling from these compact models, we can conditionally regenerate meaningful samples from computed clusters. This is known as unsupervised conditional image generation (Hwang et al. 202T
To cluster features, we exploit the fact that the rate reduction framework ① is inspired by unsupervised clustering via compression (Ma et al., 22007), which provides a principled way find the membership II. Concretely, we maximize the same rate reduction objective (I) over II, but fix the learned representation Z instead. We simply view the membership II as a nonlinear function of the features Z, say hn(( {): Z - II with parameters Ç. In practice, we model this function with a simple neural network, such as an MLP head right after the output feature: z. To estimate a "pseudo" membership I' of the samples, we solve the following optimization problem over II:
arg max max AR(2) Z (E)).
(10)
Experiments in Section 4.2 demonstrate that conditional image generation from clusters produced in this manner result in high-quality images that are highly similar in style.
4 EXPERIMENTS
We now evaluate the performance of the proposed U-CTRL framework and compare it with represen- tative unsupervised generative and discriminative methods. The first set of experiments (Section 4.] show that despite being a generative method in nature, U-CTRL can learn discriminative representa- tions competitive with state-of-the-art discriminative methods. The second set (Section 4.2 show that the learned generative representation can significantly boost the performance of unsupervised conditional image generation. Finally, the third set (Section study the advantages that generative represeentations have over discriminative ones.
We conduct experiments on the following datasets: CIFAR-10 Krizhevsky et al. 2014), CIFAR-100 (Krizhevsky et al.) 2009), and Tiny ImageNet (Deng et al., 2009). Standard augmentations for self-supervised learning are used across all datasets (Chen et al. 2020b).
We design all experiments to ensure that comparisons against U-CTRL are fair. For all methods that we compare against, we ensure that experiments are conducted with similar model sizes. If code for similar size structure can not be found, we uniformly use ResNet- 18 to reproduce results for basclines, which is larger than the network used by our method. Details about network architectures and the experimental setting are given in Appendix A All methods have runned 400 epochs or equivalent iterations (because generative models often count in iteration).
Notice that computing the rate reduction terms AR for all samples or batch of samples requires computing the expensive log det of large matrices. In practice, from the geometric meaning of R for two vectors, AR be approximated with e? norm or the cosine distance hetween two vectors.

---

Under review as a conference paper at ICLR 2023
CIFAR-10
CIFAR-100
Tiny-ImageNet
Method
Accuracy
Accuracy
Accuracy
GAN based methods
SSGAN-L (Hou et al. 2021
0.803
0.543
0.344
DAGAN+ Antoniou et al. 2017
0.772
0.519
0.224
ContraD & Shin
2021)
0.852
0,514
VAE based methods
PATCH-VAE (Parmar et al.
2021
0,471
0.325
3 3-VAE (Higgins et al. 2016
0.531
0.315
CTRL based methods
STRL-Binary(Dai al.
2022
0.599
U-CTRL (ours)
0.874
0.552
0.360
Table 2: Comparison of classification accuracy CIFAR-10, CIFAR-100, and Tiny- Imy-ImmgeNet with other generative self self-supervised learning methods. U-CTRL is clearly better.
CIFAR-10 10
CIFAR 100
Tiny-ImageNe
Method
Accuracy
Accuracy
Accuracy
SIMCLR
0.869
0,545
0,359
MoCoV2
0.872
0.589
0,365
BYOL
0.883
0,581
0,371
U-CTRL (ours)
0.874
0,552
0.360
Table 3: Comparison of classification accuracy on CIFAR-10, CIFAR-100, and Tiny-ImageNNet with purely discriminative self-supervised learning methods. U-CTRL is close to these non-generative methods.
4.1 DISCRIMINATIVE QUALITY OF LEARNED REPRESENTATIONS
To evaluate the discriminative quality of the learned representations, we follow the standard practice of evaluating the accuracy of a simple linear classifier trained on the learned representation. Table 2 compares our method against SOTA generative self-supervised learning methods, and Table compares our method against SOTA discriminative self-supervised methods. Experimental and training details given in Appendix A
Quantitative Comparisons of Classification Performance. From Table 2 we observe that on all
chosen datasets, our method achieves substantial improvements compared to existing generative self-supervised learning methods. This includes more complex datasets like CIFAR-100 100 and Tiny- ImageNet, where surpass the current SOTA models. From Table 3 our method achieves similar performance compared to SOTA discriminative self-supervised models. These results echo our goal of seeking more unifed generative and discriminative representations: despite resembling generative method architecturally, our method still produces highly discriminative representations. In addition, these results lead us to ask fundamental question: when is incorporating both discriminative and generative properties greater than seperately handling these two parts? We provide preliminary answers in Section 4.3
Qualitative Visualization of Learned Representations. To explain the classification performance of our method, we visualize the incoherence between features learned for the training datasets. Figure 2 shows cosine similarity heatmaps between the learned features, organized by ground-truth class labels. A block- bock-diagonal pattern emerges automatically from U-CTRL training for all three datasets, similar those observed in features learned a supervised setting (Dai al. (202)). this case, however, these blocks emerge and correspond with classes labels despite the absence of any supervision at all.
4.2 IMPROVED UNSUPERVISED CONDITIONAL GENERATION QUALITY
To evaluate the quality of unsupervised conditional image generation, measure performance on two axes: cluster quality and image quality. We estimate clusters by optimizing (10). and show results and comparisons with both recent and classical methods in Table Training details of our method for the additional MLP head can found the Appendix A
Cluster Quality. We measure normalized mutual information (NMI) and clustering accuracy for cluster quality on CIFAR-10 clustered into 10 classes and CIFAR- CIFAR-100(20), which clustered into 20 super-classes. From Table 4 we observe that CIFAR-10, U-CTRL results in NMI that is almost

---

Under review as a conference paper at ICLR 2023
170
(a) CIFAR-10
(b) CIFAR-100
(c) Tiny ImageNet
Figure 2: Emergence of block-diagonal structures of Z Z in the feature space for CIFAR-10 10 (left), 10
random classes from CIFAR- 100 (middle), and random classes from Tiny ImageNet (right).
(a) CIFAR-10 X
(b) CIFAR-10
Figure 3: Sample-wise self-consistency: visualization of images X and reconstructed X CIFAR-10 dataset.
double that of the existing SOTA on both GAN-based and VAE-based methods, with significantly improved clustering accuracy. Unlike many baselines, we also demonstrate that method scales to the more challenging CIFAR-100(2)) dataset, where it also significantly outperforms alternatives. Our improved clustering quality suggests potential for improving unsupervised conditional image generation, which relies on first finding statistically (and hence visually) meaningful clusters.
Image Quality. We use Frechet Inception Distance (FID) (Heusel al. 2017) and Inception Score (IS) (Salimans et al.] (2166) to measure image quality. From Table it is evident that U-CTRL maintains competitive image quality compared to other methods, measured both by FID and IS. We also compare original images against reconstructed ones in Figure 3 where we see that the original X is very similar to the reconstructed X; U-CTRL indeed achieves very good sample-wisg wise self-consistency.
Unsupervised Conditional Image Generation. In Figure 4 visualize images generated from the ten unsupervised clusters from (10). Each block represents one cluster and each row represents one principal component for each cluster. Despite learning and training without labels, the model not only organizes samples into correct clusters, but is also able to preserve statistical diversities within each cluster/class. We can easily recover the diversity within each cluster by computing different principal components and then sample and generate accordingly! More detailed illustrations with more samples is provided in Appendix B
4.3 BENEFITS OF U-CTRL'S 's STRUCTURED REPRESENTATION
As shown in the previous section, on datasets like CIFAR-10, 10, CIFAR- 100, and Tiny- ImageNet, framework is able to achieve representation quality close with the best discriminative self- supervised learning methods. clear advantage of this is computational efficiency; only a single representation needs to be trained for much broader of tasks. This subsection aims provide additional insights on how a unified model can be more beneficial for a broader range of tasks.
Domain Transfer. Regenerating images demanding on the encoder, which required to produce more informative representation than contrastive training would. We hypothesize that the encoder trained with generative task may retain more information about the image and allow the representation to generalize better. To verify this, we compare the accuracy on CIFAR- 100 using models learned from CIFAR- 10 Table 5 When compared to purely discriminative self-supervised learning models, we observe that U-CTRI. is 4 percent better than other methods on classification accuracy.
Visualization of Emerged Structures. The representations learned by U-CTRL are significantly different from those learned from previous either discriminative and generative methods. To illustrate this, we use t-SNE (Van der Maaten & Hinton 2008) to visualize the learned representation in 2D. Figure compares the t-SNE of representations learned for CIFAR- 10 by U-CTRL and MoCoV2,
8

---

Under review as a conference paper at ICLR 2023
Figure 4: Unsupervised conditional image generation from each cluster CIFAR-IO. 10, using U-CTRL. Images from different rows mean generation from different principal components of each cluster.
CIFAR-10
CIFAR- 100(20)
Method
NMI
Accuracy
FIDL IST
NMI Accuracy
FID4
1ST
GAN based methods
Self-Conditioned GAN (Liu et al.
2020
0.333
0.117
18.0
7.7
0.214
0.092
24.1
5.2
SI SLOGAN (Hwang
2021
0.340
20.6
VAΣ based methods
GMVVE(Dilkthhanaal
2016
0.247
Variational Clustering
0.445
CTRL based methods
U-CTRL (ours)
0.658
0.799
17.4 8.1 0.374 0.433
20.1
7.7
Table 4: Comparison of quality UCIG CIFAR-I0 and CIFAR-100(20), Many of methods compared do not provide code that scales up to CIFAR-100(2)) in which case we leave the corresponding table cell blank. respectively. It is clear that the representation learned by U-CTRL highly structured and organized: classes are more evident, and features within each class form clear piecewise linear structures. We present more t-SNE comparisons in Appendix E
Method
SIMCLR
MoCoV2
BYOL
U-CTRL
Accuracy
0.422
0.436
0.437
0.481
Table 5: Comparing the transfer ability with purely discriminative self-xupervised learning methods. methods are trained unsupervised on CIFAR-10 10 and tested on CIFAR-100.
(a) U-CTRL
(b) McCoV2
Figure 5: t-SNE visualizations of learned features of CIFAR-10 with different models.
CONCLUSION AND DISCUSSION
In this work, we proposed unsupervised formulation of the closed-loop transcription frame- work (Dai et al., 2022). We experimentally demonstrate that it is possible to learn a unified represen- tation for both discriminative and generative purposes, resulting in highly structured representations. Further, we show that these two purposes mutually benefit each other in various tasks, c.g., conditional image generation and domain tranfers. Compared to the more specialized representations learned in prior works, our results suggest that such unified representation has the potential in supporting and benefiting a wider range of new tasks. future work, we believe the learned representations can be further improved by jointly optimizing the feature representation and feature clusters, suggested in the original rate reduction paper (Chan et al. 2022). Features with high likelihood of belonging to the same cluster be further linearized and compressed. Due to its unifying nature and the simplicity of the underlying concepts, this new framework may be extended beyond image data, such sequential dynamical observations.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
All authors agree and will adhere to the conference's Code of Ethics. We do not anticipate any potential ethics issues regarding the research conducted in this work.
REPRODUCIBILITY STATEMENT
Settings and implementation details of network architectures, optimization methods, and some common syper-parameters described in the Appendix A We will also make source code available upon request by the reviewers or the area chairs.
REFERENCES
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arXiv:1711.04344 2017.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Vaniamee-ivarrinnc--ovaiaa regularization for self-supervised learning. arXiv preprint arXiv:2105.04900 2021.
Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi. John Wright, and Yi Ma. ReduNet: A white-box deep network from the principle of maximizing rate reduction. Journal of Ma- chine Learning Research, 23(114):1-103 2022. URL http:/htt..... org/papers/v233
21 0631. html
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691- 1703. PMLR, 2020a.
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby, Self-supervised gans via auxiliary rotation loss. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 2215--1213,, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representatioon learning. arXiv preprint arXiv:2202.03026 2022.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2020.
Xili Dai, Shengbang Tong, Mingyang Li, Ziyang Wu, Michael Psenka, Kwan Ho Ryan Chan, Pengyuan Zhai, Yaodong Yu, Xiaojun Yuan, Heung- Beung-Yeung Shum, et al. Ctrl: Closed-loop transcription to ldr via minimaxing rate reduction. Entropy, 24(4):456, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Nat Dilokthanakul, Pedro AM Mediano, Marta Gareel,, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders. arXiv preprint arXiv:/61 1.02648. 2016.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint arXiv: 1606:00900782 2016.
10

---

Under review as a conference paper at ICLR 2023
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv:1606.00704, 2016.
William Falcon, Ananya Harsh Jha, Teddy Koker, and Kyunghyun Cho. Aavae: Augmentation-
augmented variational autoencoders. arXiv preprint arXiv:2107. 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances Neural Information Processing Systems, 33:21271 1-21284, 2020a.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In NeurlPS, 2020b.
Kamal Gupta, Saurabh Singh, and Abhinav Shrivastava. Patchvae: Learning local latent codes for recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 475544475, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738. 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. arXiv preprint urXiv:2111.0633 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. arXiv preprint arXiv: 03599, 2016.
Liang Hou, Huawei Shen, Qi Cao, and Xueqi Cheng. Self-supervised gans with label augmentation. Advances in Neural Informtionn Processing Systems, 34, 2021.
Uiwon Hwang, Heeseung Kim, Dahuin Jung, Hyemi Jang, Hyungyu Lee, and Sungroh Yoon. Stein latent optimization for generative adversarial networks. arXiv preprint arXiv:2106.053199 2021.
Jongheon Jeong and Jinwoo Shin. Training gans with stronger augmentations via contrastive discriminator. arXiv preprint urXiv:2103.09742 2021.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embed- ding: An unsupervised and generative approach to clustering, arXiv preprint 1rXiv:1611.0511 2016.
Sheena A. Josselyn and Susumu Tonegawa. Memory engrams: Recalling the past and imagining the future. Science, 367, 2020.
Georg B Keller and Thomas D Mrsic-Flogel. Predictive processing: A canonical cortical computation. Neuron, 100(2):424-435, October 2018.
Sachoon Kim, Sungwoong Kim, and Juho Lec. Hybrid generative-contrasivve representation learning. arXiv preprint orX1v22106.06162, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 441266980, 2014.
11

---

Under review as a conference paper at ICLR 2023
Diederik P Kingma and Max Welling. arXiv:1312.611,, 2013.
Auto-encoding variational bayes.
arXiv preprint
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 55, 2014.
The CIFAR-10 dataset.
online:
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. arXiv preprint arXiv: 14, 2009.
Zengyi Li, Yubei Chen, Yann LeCun, and Friedrich T Sommer. Neural manifold clustering and embedding. arXiv preprint wrXiv:2201.10000 2022.
Steven Liu, Tongzhou Wang, David Bau, Jun- Yan Zhu, and Antonio Torralba. Diverse image generation via self-conditioned gans. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 14286-14295, 2020.
Yi Ma, Harm Derksen, Wei Hong, and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. PAMI, 2007.
Sudipto Mukherjee, Himanshu Asnani, Eugene Lin, and Sreeram Kannan. Clustergan: Latent space clustering in generative adversarial networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 4610-4611 2019.
Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative autoencoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 823-832, 2021.
Vignesh Prasad, Dipanjan Das, and Brojeshwar Bhowmick. Variational clustering: Leveraging variational autoencoders for image clustering. In 2020 International Joint Conference on Neural Networks (IJCNN, pp. 1-10. IEEE, 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 151:166311,4 2015.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, Advances in neural information processing systems, 29, 2016.
Shengbang Tong, Xili Dai, Ziyang Wu, Mingyang Li, Brent Yi, and Yi Ma. Incremental learning of structured memory via closed-loop transcription. arXiv:2202.05411, 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Gido M Ven, Hava Siegelmann, Andreas S Tolias, al. Brain-inspired replay for continual learning with artificial neural networks. Nature Communications, 11(1):1-14, 2020.
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004. doi: 10.1109/T1P220008889999
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 0019422294344 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
12

---

Under review as a conference paper at ICLR 2023
A TRAINING DETAILS
A.1 NETWORK ARCHITECTURES
Table and Figure 6 give details on the network architecture for the decoder and the encoder networks used for experiments. The black rectangle marked with "conv, s=2" means a convlutional layer with stride The orange rectangle marked with "dconv, s=2" means deconvolutional layer with stride 2. The "x k" besides red frame means we regard these layers in red frame block and stack times.
a values in Leaky-ReLU (i.e. IReLU) of the encoder are set to 0.2. We set (nz 128, nc 3. k = 3) for CIFAR-10, (nz 256,
3, k 4) for CIFAR-100 100, and (n2 = 256, nc = 3, = 4) for Tmmy-ImageNet. As a comparison, ResNet- 18 contains around 11 million parameters, whereas encoder only contains between 4 and million parameters depending on the choice of k.
Table gives details of the network architecture for the linear classifier and Table gives details of the network architecture for the additional MLP head used for unsupervised conditional image generation training.
R 1
ResBlock Up. 256
ResBlockUp. 128
ResBlockUp. 64
4 X 4, stride=2, pad=1 deconv. 1 Tanh
Table 6: Network architecture of the decoder g(,, 7))
Image E 32x322m
ResBlockDown 64
ResBlockDown 128
ResBlockDown 256
4 x 4, stride=1, pad=0 conv 722
Table Network architecture of the encoder (-,0).
Dcony, s-2
conv, s=2
BN
BN
Relu
L-Relu
cony,
conv, s=1
BN
x k
BN
x
Relu
L-Relu
ResBlock Up architecture
ResBlock Down architecture
Figure 6: Architecture of two ResBlock.
A.2 OPTIMIZATION
For all experiments, 3 = 0.5, ,3 = 0.999. The learning rate is set to be 0.000T. We choose è2 0.2. For all experiments,
Adam (Kingma & Ba 2014)
as optimizer, with hyperparameters
adopt augmentation from SimCLR (Chen 2020b
13

---

Under review as a conference paper at ICLR 2023
R 1x
Linear(nz, number of class)
Table 8: Network architecture of the linear classifier.
R Ix
Linear(nz, nz) ReLU
Linear(nz, number of clusters)
Table Network architecture of the MLP head for unsupervised conditional image generation
For CIFAR-10, 10, CIFAR-100, and Tiny ImageNet, train our framework with batch size of 1024 over 20,000 iterations. All experiments are conducted with at most 4 RTX 3090 GPUs. Methods that compared against in Table 3 are trained with the batch size of 256, because Chen al. (2020b) observe that purely discriminative methods tend to perform better with smaller batch sizes. Table methods have used their optimal parameters in their github code.
For training of the MLP head for unsupervised conditional image generation 10p, we again use Adam (Kingma & Ba 2014) as our optimizer with hyperparameters 31 0.5, B2 0.999. We choose the learning rate to be 0.0001 and e² as 0.2, with batch size 1024 over 5000 iterations.
For training of the linear classifier, we use Adam (Kingma Ba, 2014 as our optimizer with hyperparameters 81 = 0.5, B2 = 0.999. We choose learning rate to be 0.0001, with batch size 1024 over 5000 iterations.
B ADDITIONAL UNSUPERVISED CLUSTERING AND GENERATION RESULTS
B.1 CLUSTER RECONSTRUCTION
In this subsection, we visualize the reconstruction of ten clusters that are predicted and generated by U-CTRL on the CIFAR-10 10 training set. Each block in Figure contains both a random sample of reconstructed data in a cluster and the total number samples within it. Note that CIFAR-IU 10 contains 50,000 training samples, split across 10 classes. As we see in Figure the number of samples in each cluster are very close to 5,000, with the largest deviator (cluster 9) containing 3,942 samples. Without any cues,
easily identify correspond each unsupervised cluster with a CIFAR- 10 class. For a class like 'bird', we observe that the model is able to group images of standing birds, flying birds, and bird heads, despite their visual differences.
B.2 UNSUPERVISED CONDITIONAL IMAGE GENERATION
Building U-CTRL's 's ability to cluster CIFAR- 10 samples, demonstrate the model's ability to perform unsupervised conditional image generation Figure In contrast to reconstruction, where images regenerated from features corresponding real samples, generate images based the feature sampling technique proposed in (Dai et al. (022). From these results, we observe that the U-CTRL framework maintains in-cluster diversity, and that the diversity can be recovered and visualized via simple principal component analysis.
C ABLATION STUDIES
C.1 THE IMPORTANCE OF EACH TERM IN U-CTRL FORMULATION
In this section, we study the significance of the sample-wise constraints and extra rate distortion term in the formulation Table prresents the following objectives that study:
Objective I is the constrained U-CTRL maximin 7
Objective II is the constrained maximin without the augmentation compression constraint 6
14

---

Under review as a conference paper at ICLR 2023
2232 images
Custer sc50 iagge
(a) Cluster I
(b) Cluster 2
Chutter with 4003 images
Cludtrr with 5026
(c) Cluster
(d) Cluster
Clyste
(e) Cluster 5
(f) Cluster 6
Cluserwith with 5121
(g) Cluster 7
(h) Cluster
maags
(i) Cluster 9
i Cluster 10
Figure More result on the reconstruction of clusters in CIFAR-10
Objective III is the constrained maximin without the sample-wise self-consistency constraint 5 Objective IV the constrained maximin without the extra rate distortion term.
Objective the U-CTRL without the augmentation compression constraint and sample-wise
self-consistency constraint.
Objective VI the CTRL-Binary maximin formulation 4
Table 11 shows the result of linear probe for representations trained using each objective CIFAR- 10. From the table, it is evident that both constraints and the rate distortion term are pivotal to the success of framework.
Objective I:
maxx min, R(Z) AR(Z, 2) s.t. Σ
IEN AR(2', and EiEN AR(2*, 22)
Ohjective
maxo min, R(Z) AR(Z, 2) s.t. LieN AR(z', #)
Objective III:
maxe min,, R(2) AR(Z, 2) s.t. Eien AR(z', x1)
Objective IV:
min, AR(Z, 2) s.t. EiEN AR(z', =²)= and EIEN AR(
( z =
Objective maxe min, R(Z) + AR(Z,2) Z) maxe min, AR(Z, 2)
Objective VI:
Table 10: Five different objective functions for U-CTRL.
15

---

Under review as a conference paper at ICLR 2023
(a) Cluster
(b) Cluster 2
(c) Cluster
(d) Cluster
(c) Cluster 5
(f) Cluster
(g) Cluster
(h) Cluster
ii) Cluster
Cluster 10
Figure 8: Unsupervised conditional image generation on CIFAR-10 10. Each block represents a cluster, within which each row represents one principal component direction in the cluster, and samples along each row represent different noises applied in that principal direction.
C.2 THE IMPORTANCE OF MCR² IN U-CTRL FORMULATION
In this section, we verify the significance MCR² term AR Z, 2) in our method. We do ablation study CIFAR-10 with the same network and training condition. If take away MCR2 from
16

---

Under review as a conference paper at ICLR 2023
Method
Objective
Objective II Objective III
Objective IV
Objective V
Objective VI
Accuracy
0.874
0.578
0.644
0.522
0.633
0.599
Table 11: Ablation study on significance different terms U-CTRL..
formulation, it changes (T1b. For simplicity, we call it U-CTRL-noMCR* R(Z)
max min
(1I)
subject
CAR(E',E) 00 and EAR =0.
Table 12 shows that U-CTRL without the MCR not only learns worse representation but also generalizes worse to out of distribution data. Figure visualizes the reconstructed X by U-CTRL- noMCR2 It is clear from the image figure that without the MCR the decoder fails reconstruct high-quality images.
Accuracy on CIFAR-10
Transfer Accuracy on CIFAR-100
U-CTRL
0.874
0,481
U-CTRL-noMCR
0.836
0.418
Table 12: Ablation study the significance of MCR" in U-CTRL.
(a) CIFAR-10 X
(b) CIFAR-10 10 X
Figure Visualization images trained by U-CTRI-noMCR': reconstructed X on CIFAR-10 datasct.
follows our discussion the introduction that discriminative tasks and generative tasks together learn feature that benifits each other.
D RANDOM SEED SENSITIVITY
In this section, we verify the stability of our method against different random seeds. We report in Table Table the accuracy of U-CTRL CIFAR-10 with different seeds. We observe that the choice of seed has very little impact on performance.
Random Seed
5
10
15
100
Accuracy
0.874
0.876
0.870
0.874
0.871
Table 13: Ablation study on varying random seeds.
E MORE COMPARISON ON T-SNE
Due to limited space in the main body, present comparison of t-SNE between u-CTRL and other discriminative- based methods in this section. As shown in Figure [0] u-CTRL enjoys more structured representation comparing purely discriminative methods.
17

---

Under review as a conference paper at ICLR 2023
(a) U-CTRI
(b) McCoV2
(c) BYOL
(d) SimCLR
Figure 10: t-SNE visualizations of learned features of CIFAR-10 with different models.
18


Reviews:
[{"review_id": "xbKZrswBeEI", "reviewer": "Reviewer_jYoM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- This paper illustrates a learning framework toward a unified representation which is effective for both generative and discriminative purposes.\n- To do so, this paper introduce the idea of recent work, \"Closed-Loop Transcription (CTRL) via a Constrained Maximin Game\" in an unsupervised fashion.\n- Specifically, the proposed method minimize the rate reduction between $Z$ and $\\hat{Z}$, where $X \\xrightarrow{f(x,\\theta)} Z \\xrightarrow{g(z,\\eta)} \\hat{X}  \\xrightarrow{f(x,\\theta)} \\hat{Z}$, $f(x,\\theta)$ is encoder and $g(z,\\eta)$ is decoder.\n- Then, the rate reduction between two code are defined as: $\\Delta R(Z, \\hat{Z}) \\coloneqq R(Z \\cup \\hat{Z}) - \\frac{1}{2}(R(Z) + R(\\hat{Z}))$, where $R(Z)\\coloneqq\\frac{1}{2}\\log\\det(I+\\frac{d}{N\\epsilon^2}ZZ^\\top) $.\n- This paper also propose auxiliary loss term to 1) improve sample-wise self-consistency: $\\sum\\limits_{i\\in N} \\Delta R(z^i, \\hat{z^i})$, and 2) self-supervision:  $\\sum\\limits_{i\\in N} \\Delta R(z^i, \\hat{z}_a^i)$.\n- Putting all this together, the unsupervised CTRL objective is defined as follows: \n$\\max\\limits_\\theta R(Z) + \\Delta R(Z, \\hat{Z}) - \\lambda_1 \\sum\\limits_{i\\in N} \\Delta R(z^i, \\hat{z^i}) - \\lambda_2 \\sum\\limits_{i\\in N} \\Delta R(z^i, \\hat{z}_a^i)$\n- $\\min\\limits_\\eta R(Z) + \\Delta R(Z, \\hat{Z}) + \\lambda_1 \\sum\\limits_{i\\in N} \\Delta R(z^i, \\hat{z^i}) + \\lambda_2 \\sum\\limits_{i\\in N} \\Delta R(z^i, \\hat{z}_a^i)$\n- The experiments are conducted on three datasets: CIFAR10, CIFAR100, Tiny-ImageNet.\n- The proposed U-CTRL shows clear performance gains over generative self-supervised learning methods.\n- Its performance is also comparable to the discriminative self-supervised learning such as SimCLR, MOCO, BYOL.\n- Its generative performance (UGIG task) is better than other generative models.\n- By leveraging its generative objective, domain transfer performance is also better than other self-supervised methods.", "summary_of_the_review": "This paper propose U-CTRL for learning an unified representation (generative and discriminative). The empirical results are very convincing, however, an ablation study is highly recommended to find their novelty over previous work.", "strength_and_weaknesses": "Strength:\n1. The motivation is clear: one unified representation for both generative and discriminative purposes.\n2. This paper is written clearly.\n3. The empirical results are sufficiently strong enough to understand the effectiveness of the proposed method.\n\nWeakness:\n1. The proposed method is built upon the idea of CRTL-binary (i.e., $\\max\\limits_\\theta \\min\\limits_\\eta \\Delta R(Z, \\hat{Z})$). \nTo understand the contribution of this paper, ablation study of the additional loss terms (e.g., $R(Z), \\sum\\limits_{i\\in N} \\Delta R(z^i, \\hat{z^i}), \\sum\\limits_{i\\in N} \\Delta R(z^i, \\hat{z}_a^i) $  ) is highly recommended.\n2. As noticed in the footnote of page 6, $ \\Delta R(Z, \\hat{Z})$ can be replaced with $\\ell_2$ or cosine distance in practice and this trick cannot be applied to $R(Z)$ term. If rate reduction term is replaced with any simplified distances, it should be discussed in more details in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lisfEoSyCWl", "reviewer": "Reviewer_TaSr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThe authors present an unsupervised approach, based on recent ideas\ncalled 'closed-loop transcription', to extract feature encodings that\nperform well on both discriminative and generative tasks, on CIFAR-10,\nCIFAR-100 and Tiny Imagenet data sets.  Often there is a tradeoff\nbetween generative vs discriminative goals, and their approach points\nto the possibility of methods excelling at both (empirically).\n\n", "summary_of_the_review": "This is a new idea, and shows good promise in a number of directions.\nMany experiments and comparison are performed to showcase the\npotential.\n\n", "strength_and_weaknesses": "Strengths:\n\n-- A variety of experiments, on several tasks, show that the technique\n   is superior than existing unsupervised generative or discriminative\n   techniques.\n\n\nWeaknesses:\n\n-- The paper is dense, and I didn't come out with an understanding of\n   why the method may work better.\n\n-- The cost of optimization is not clear: how many rounds to\n   convergence? (for 8 and 9)\n\n\nRewordings, etc:\n\n-> (page 6, 'study how the..') Finally, the third set (Section 4.3)\nstudy how the advantages that generative representations have over\ndiscriminative ones.\n\n-> (properties a whole greater) In addition, these results lead us to\n ask a fundamental question: when is incorporating both discriminative\n and generative properties a whole greater than the sum of its parts,\n particularly outside of the context of computational efficiency?\n\n-> Why does superior clustering performance reflect the generative\n   quality of the features: it seems to be that's another way of\n   assessing the informativeness or the discriminative power of the\n   features learned.\n\n--> page 6: You mention 'structure' here and elsewhere: \"In order to\nfacilitate discriminative or generative tasks, it must be highly\nstructured.\"  Do you just mean that the features come together to\nreflect the underlying (hidden) class structure (rather than having structure\nin the sense of having sub-parts?)\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YKdUicseGi", "reviewer": "Reviewer_K48e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an unsupervised extension of the closed-loop transcription, named as U-CTRL, to learn the better representation for discriminative and generative learning tasks. The authors show that the learned representation is more structured, i.e. having the cluster shapes in semantically-meaningful way, improving the performance of linear evaluation and conditional generative tasks. ", "summary_of_the_review": "My initial evaluation is borderline, slightly leaning towards rejection. My major concern on this work is the insufficient empirical justification. In specific, I’m not sure that the learned representation based on U-CTRL is discriminative enough compared to the similar recent works. In addition, I failed to figure out that the representation learned by U-CTRL is more structured than the ones by contrastive losses. ", "strength_and_weaknesses": "**Strengths**:\n\n* This manuscript is basically very well-written, and clearly summarizes the necessary previous works to understand the main argument. It's easy to follow most of the parts in the manuscript. \n* It seems that the approach based on closed-loop transcription and rate-distortion theory for learning better representation is quite principal. \n* This manuscript includes all the details in the experiments, including detailed setup, hyperparameters, and network configuration. It seems that all the results are reproducible. \n\n\n**Weaknesses**: \n\nMy biggest concern on this work is the low performance of discriminative learning shown in Table 2. Compared to the similar baselines, Table 2 is not enough to argue that the learned representation by U-CTRL is highly discriminative. I extracted the linear evaluation experiments in GCRL and W-MSE, as shown in below:\n\n\n|Method| CIFAR-10 | CIFAR-100 | Tiny ImageNet|\n|----------|----------|----------|----------|\n|U-CTRL| 87.4 | 55.2 | 36.0 | \n|GCRL | 83.9-95.1 | 58.7 - 76.0 | - | \n|W-MSE | 91.6 - 92.0 | 66.1 - 67.6 | 48.2 - 49.2 |\n\nBesides the comparison to GCRL and W-MSE, I’ve found that the performance of BYOL in Table 2 is much worse than the reported numbers in Table 1 in W-MSE paper. Could you explain a little bit more why the performance gap is observed? \n\nI’m also still not convinced that the learned representation based on U-CTRL is more structured than SSL methods, especially the methods based on contrastive losses. Therefore, it could be necessary to measure the cluster quality of representation learned by contrastive losses, since the contrastive loss enforces the semantically-meaningful clusters in the representation space. In this work, U-CTRL was only compared to the learned representation based on generative learning, which is not sufficient to argue that the representation from U-CTRL is more structured than the ones from existing approaches. Figure 5 seems to support the argument, but it could be better to include the results of SimCLR and BYOL. \n\nThis work also argues that the structured representation improves the quality of generated samples a lot. However, as shown in unCLIP (a.k.a. DALL-E 2), the representation based on a simple contrastive loss definitely helps to train a decoder. It means that the learned representation based on contrastive losses is already structured enough to train a generative model.\n\nReferences:\n* [W-MSE] Whitening for Self-Supervised Representation Learning, ICML’21. \n* [GCRL] Hybrid Generative-Contrastive Representation LEarning, arXiv’21.\n* [unCLIP] Hierarchical Text-Conditional Image Generation with CLIP Latents, arXiv’22.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Wz_6kAqF4-", "reviewer": "Reviewer_Q2mk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper claims to extend the previously proposed CTRL framework for representation learning to unsupervised setting. Paper proposes self-consistency and augmented sample compression based constraints to define an objective function for representation learning. Several experiments evaluate the representations learned by the proposed method by comparing against both discriminative and generative prior art.", "summary_of_the_review": "I am leaning reject as the contributions seem marginal at best. Further, method is behind in terms of learned representation quality in comparison to discriminative methods. In addition, there are several other issues as detailed in the weaknesses section.", "strength_and_weaknesses": "Strengths\n- Strong performance against generative baselines, although on smaller datasets.\n\nWeaknesses\n- Unclear contribution: While the paper explicitly claims that it extends the CTRL framework to unsupervised setting, eq 4 already appears in Dai et al., 2022. Unless I am mistaken, CTRL framework as in Dai et al, 2022 is not limited to supervised setting only. This fact dilutes the claimed contribution. However, the incorporation of constraints (eq 5, 6) appears to be new.\n- Limited Novelty: It seems that incorporation of self-consistency (eq 5) and augmentation (eq 6) are the main contributions of the paper. These on their own seem to be marginal at best. While experiments show that these do lead to superior results, some crucial ablations are missing (see next pt.)\n- Missing ablations: Given that constraints listed in eq 5 and eq6 are the main contribution, these choices need to be studied. More specifically, how does the performance change from no constraint, to only first constraints, to only second constraint, to both of them?\n- Inferiority to \"discriminative\" methods: The methods categorized under discriminative do not use labels (just like the proposed method), and appear to outperform it on the evaluated datasets. Further, more recent methods e.g. DINO [1], are superior to these baselines, while using a \"simpler\" training objective in comparison to the proposed method. This further reduces the appeal of the proposed method.\n- Missing larger scale experiments: Most recent representation learning methods report their performance on the larger ImageNet dataset, to more closely mimic the practical setting. It would be beneficial to have that comparison to place the proposed work w.r.t. to these methods.\n\n\n\n[1} Emerging Properties in Self-Supervised Vision Transformers, Caron et al.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject