Structural Adversarial Objectives for Self-Supervised Representation Learning
Authors: Xiao Zhang
Michael Maire
Venue: Submitted to ICLR 2023 (2023)

Abstract: Within the framework of generative adversarial networks (GANs), we propose objectives that task the discriminator with additional structural modeling responsibilities.  In combination with an efficient smoothness regularizer imposed on the network, these objectives guide the discriminator to learn to extract informative representations, while maintaining a generator capable of sampling from the domain.  Specifically, we influence the features produced by the discriminator at two levels of granularity.  At coarse scale, we impose a Gaussian assumption encouraging smoothness and diversified representation, while at finer scale, we group features forming local clusters.  Experiments demonstrate that augmenting GANs with these self-supervised objectives suffices to produce discriminators which, evaluated in terms of representation learning, compete with networks trained by state-of-the-art contrastive approaches.  Furthermore, operating within the GAN framework frees our system from the reliance on data augmentation schemes that is prevalent across purely contrastive representation learning methods.

Under review as a conference paper at ICLR 2023
STRUCTURAL ADVERSARIAL OBJECTIVES FOR SELF-SUPERVISED REPRESENTATION LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Within the framework of generative adversarial networks (GANs), we propose objectives that task the discriminator with additional structural modeling respon- sibilities. In combination with an efficient smoothness regularizer imposed on the network, these objectives guide the discriminator to learn to extract informative representations, while maintaining generator capable of sampling from the do- main. Specifically, we influence the features produced by the discriminator at two levels of granularity. At coarse scale, impose Gaussian assumption encour- aging smoothness and diversified representation, while at finer scale, we group features forming local clusters. Experiments demonstrate that augmenting GANs with these self-supervised objectives suffices to produce discriminators which, evaluated in terms of representation learning, compete with networks trained by state-of-thee art contrastive approaches. Furthermore, operating within the GAN framework frees our system from the reliance on data augmentation schemes that is prevalent across purely contrastive representation learning methods.
INTRODUCTION
Unsupervised feature learning algorithms aim to directly learn representations from data without reliance on annotations, and have become crucial efforts scale vision and language models toward handling real-world complexity. Many state-of-the-ar approaches adopt a contrastive self- supervised framework, wherein deep neural network is tasked with mapping augmented views of a single example to nearby positions in high-dimension embedding space, while separating embeddings of different examples (Wu et al., 2018; He et al., 2020; Chen et al., 2020; Chen & He, 2021; Grill et al., 2020; Zbontar et al., 2021). Though requiring no annotation, and hence unaffected by assumptions baked into any labeling procedure, the invariances learned by these models are still influenced by human-designee heuristic procedures for creating augmented views.
The recent prominence of contrastive approaches was both preceded by and continues alongside a focus engineering Aomain-relevan, proxy tasks for self-supervised learning. For computer vision, examples include learning geometric layout (Doersch et al., 2015), colorization (Zhang et al., 2016; Larsson et al., 2017), and inpainting (Pathak et al., 2016; He al., 2022). Basing task design domain knowledge may prove effective in increasing learning efficiency, but strays further from an alternative goal of developing truly general and widely applicable unsupervised learning techniques.
A third family of approaches, coupling data generation with representation learning, may provide a path toward such generality while also escaping dependence upon the hand-crafted elements guid- ing data augmentation or proxy task design. Generative adversarial networks (GANs) (Goodfellow al., 2020) and variational autoencoders (VAEs) (Kingma & Welling, 2013) are prime examples within this family. Considering GANs, one might expect the discriminator to act as an unsuper- vised representation learner, driven by the need to model the real data distribution in order to score the generator's s output. Indeed, prior work finds that some degree of representation learning occurs within discriminators in a standard GAN framework (Radford et al., 2015). Yet, to improve genera- tor output quality, limiting the capacity of the discriminator appears advantageous (Arjovsky et al., 2017) a choice potentially in conflict with representation learning. Augmenting the standard GAN framework to separate encoding and discrimination responsibility into different components (Don- ahue et al., 2017; Dumoulin et al., 2017), along with scaling to larger models (Donahue & Simonyan, 2019), is promising path to circumventing this apparent limitation.

---

Under review as a conference paper at ICLR 2023
Generative Adversarial Network
Structural objectives
(1 ED
D
3
10
(12 Σ②)
D
Gaussian
Visualize embedding with t-SNE
Cluster
atrilane
Legend
D attraction
diee
repuising
Bog
attraccton
horse
Stp
treE
Figure 1: Within the GAN framework, we illustrate our proposed structural adversarial objectives, which push the discriminator learn good feature representations. We combine objectives from two levels of granularity. Top right: At coarse scale, we model features under the Gaussian assumption, and adversarially orient generated embedding (orange) towards the real embedding (blue), using the sample mean p and covariance E. Bottom right: At a finer scale, we implement clustering objectives, which group adjacent real embeddings to form a cluster and adversarially attract the generated embedding towards the nearby cluster center. Bottom left: Learned representations on CIFAR-10 show a distribution consistent with categorization.
However, it has been unclear whether the struggle to utilize vanilla GANs as effective representation learners stems from inherent limitations of the framework. We provide evidence the contrary, through an approach that significantly improves representations learned by the discriminator, while maintaining generation quality and operating with standard pairing of generator and discriminator components. Our approach only modifies the training objectives within the GAN framework, with two aims: (1) regularize the smoothness of the discriminator while maintaining adequate capacity, and (2) require the discriminator model additional structure of the real data distribution.
To control discriminator smoothness, we propose an efficient regularization scheme that approxi- mates the spectral norm of the Jacobian. Specifically, it instantiates the matrix-vectoe subroutine in a power iteration with an efficient vector-Jacobian product protocol, preventing the need to compute the full Jacobian matrix of the neural network.
To prompt the discriminator to learn embeddings that model the data distribution, we propose ad- versarial objectives resembling contrastive self-supervised clustering target. These objectives in- fluence the output of the discriminator at two levels of granularity. At a coarse level, we make a Gaussian assumption aiming for diversified and smooth representation. At more refined scale, adopt a mean shift clustering objective to group features. Though imposing attractive and repulsive forces onto embeddings produced by the discriminator (Figure 1), do not rely on data augmenta- tion to drive learning. Instead, both real and fake data participate in an adversarial clustering game.
Experiments on representation learning benchmarks show our method achieves competitive perfor- mance with recent sate-of-the-art contrastive self-supervised learning approaches, even though we do not leverage information from (or even have concept of) an augmented view. We also demon- strate that supplementing GAN with our proposed objectives not only enhances the discriminator as representation learner, but also improves the quality of samples produced by the generator.
RELATED WORK
GENERATIVE FEATURE LEARNING
Much research GANs has focused improving the quality of generated data, yielding signifi- cant recent advances (Karras et al., 2017; 2019; 2020; 2021; Sauer et al., 2022). Other efforts have focused evolving capabilities, including conditional and controllable generation, egg. text im-
2

---

Under review as a conference paper at ICLR 2023
age generation (Zhang et al., 2021; Hinz al., 2020) and regmentation-uuidee image generation (Zhu et al., 2017: Chen & Koltun, 2017). In comparison, adopting GANs for unsupervised fea- ture learning has been relative scareely explored. In this area, adversarial approach dependent upon an additional encoder component (Donahue et al., 2017; Dumoulin et al., 2017; Donahue & Simonyan, 2019) appears most successful to date. Here, the encoder is tasked with learning to invert the function of the generator, mapping data to a latent embedding space. The discriminator then acts on (data, latent) pairs, and is tasked with distinguishing between real data paired with the out- put of the encoder run on it, or vice-versa for sampled latent and corresponding generator output. Representation learning is the responsibility of the encoder, rather than the discriminator.
Besides GANs, other generative models also demonstrate feature learning capacity: Zhang et al. (2022); Ma et al. (2021) propose discard the low-level structures in VAE and Flow models to improve learned representations. Du et al. (2021) show that an unsupervised energy model can learn semantic structures, e.g., segmentation and viewpoint, from images. Preechakul et al. (2022) attach an encoder diffusion model and show that it learns high-level feature representations.
We adopt an orthogonal approach that, by imposing structural adversarial objectives in GAN's training, tasks the discriminator to learn richer data representations. Dai et al. (2022) introduce sim- ilar objectives in their closed-loop control pipeline, where they replace the pixel-wise reconstruction target of autoencoder framework with adversarial MCR objectives.
2.2 CONTRASTIVE SELF-SUPERVISED REPRESENTATION LEARNING
In contrast to the generative approaches, contrastive self-supe learning has witnessed substan- tial progress across multiple domains, sometimes achieving nearly the same performance level as a supervised counterpart. Among them, a popular choice is to learn augmentation invariant represen- tation: Wu et al. (2018); He et al. (2020); Chen et al. (2020); Oord et al. (2018) leverage a siamese pipeline to optimize InfoNCE objective, which aims to maximize the feature similarity across augmented views, while repulsing from all other instance to maintain feature uniformity. Chen & He (2021); Grill al. (2020) further simplifies the pipeline by dropping the negative terms and leveraging specific architectural designs to prevent collapsed solutions. As an alternative to operat- ing on an 12 normalized embedding. Caron et al. (2020; 2021); Wang al. (2021) enforce clustering consistency across views. Inspired by masked language modeling, He et al. (2022); Bao et al. (2021) propose a variant in the image domain by tasking an autoencoder to predict the masked pixels.
Though contrastive approaches yield strong models in multiple benchmarks, Tian et al. (2020) show- case the limitations of view-invariant assumptions and demonstrates their sensitivity to the parame- ters of augmentation schemes. This implies that the choice of augmentation represents assumptions placed on the data distribution, complicating generalization across multiple data sources. This phe- nomenon can be observed in the scaling experiment of He et al. (2020), where increasing training images from I million (2020) raise a concern with applying those methods to broader and unconstrained datasets, where multiple object instances from the same image should not have mutually invariant representations.
billion leads to only marginal feature improvement. Zhang & Maire
2.3 STABILIZING GAN TRAINING
Despite the ability generate high-quality samples, successfully training GANs remains challeng- ing due to the adversarial optimization. To stabilize the training and scale to larger models, several approaches have been proposed. Heusel et al. (2017) suggest maintaining a separate learning rate for the generator and discriminator to maintain local Nash equilibrium. Arjovsky et al. (2017); Gulrajani et al. (2017) consider constraining the discriminator's Lipschitz constant with gradient clipping and gradient norm penalization. In contrast to regularizing model-wise functionality, Miy- ato et al. (2018) implement layer-wise spectral normalization schemes by dividing parameters with their leading singular value, which is widely adopted in recent state-of-the-art models. Wu et al. (2021); Bhaskara et al. (2022) instead propose to build a Lipschitz- constrained function by dividing the output with gradient norm and show it can preserve the model capacity. However, none of these methods suit our case, since spectral normalization (Miyato et al., 2018) harms model capacity, and gradient-based regularization only works for scalar output, limiting use of structural objectives.
3

---

Under review as a conference paper at ICLR 2023
3
METHOD
3.1 GAN PRELIMINARIES
Generative adversarial networks (GANs) include two learnable modules: generator G producing synthetic data from a latent sample z, and discriminator D, which learns to differentiate between the true data and generated samples G(z). During training, G and D are alternatively updated in an adversarial fashion, which can be formulated as a minimax problem (Goodfellow et al., 2020):
min max Em-p(a)) (log [(=)] B=>g(=) 1 log D())..
(1)
& D
There are several popular alternatives for Eqn.1, e.g. hinge loss(Lim Ye, 2017). Wasserstein distance(Arjovsse et al., 2017), and maximum mean discrepancy (MMD) (Li et al., 2017), but none provide explicit structural control of the output embeddings, i.e., the features produced by D. We propose an explicit objective to obtain structural embeddings such that, besides generative power, well-trained discriminator D accomplishes representation learning.
3.2 ADVERSARIAL LEARNING WITH STRUCTURAL OBJECTIVES
Our goal is to task D as the feature extractor to learn data representation real images. We denote the p-dimensional output from D of real images and fake generations as f. f respectively. Here f E S are normalized and live in unit hypersphere. We also maintain the unnormalized counterparts f and fa of f and f", where we explore their utility in Section 3.3.
Our proposed structural objectives operate hierarchically and regularize the learned embeddings at two levels of granularity: (1) At coarse level, assume both f, f follow Gaussian distributions: f ~ (N f ~ Nuffs fo where Jy: f4f= Mf E IRP and f Ey= E RPXP are sample mean and covariance respectively. (2) At refined level, we focus reorganizing embeddings by constructing clusters using local affinity. Therefore, by distributing our objective function with dedicated geometrie patterns, we further aim to reinforce the feature learning capability of D under the purely unsupervised setting.
Coarse-scale Optimization with Gaussian. Under unimodal Gaussian assumptions, we can si- multaneously control the geometry and locations of feature distributions using simple empirical statistics, iee, sample mean and covariance. With a selected metric d(-) for Gaussian distributions, we define our goal as minimax problem:
LGaussian == min max ff
One widely adopted candidate for d(-) is Jensen-Shannon divergence (JSD) due to its symmetry and stability. For two arbitrary probability distributions P, P,Q, JSD admits the following form:
P (Ri.(P)) (PI) + Q (QI
P P+ - H(
P Q 1 - + HTQQ)), (3)
JSD( (SD(P))Q)= DKL.
where DKL. H denotes Kullback-Leibler divergence and entropy respectively. We can compute entropy for Q, P using closed-form formulas. However, entropy for (P + Q)/2 is problematic to compute exactly and generally requires Monte Carlo simulation, an infeasible computational approach in high dimensional space. To tackle this problem, we follow (Hershey & Olsen, 2007) to
approximate P+Q by a single Gaussian and estimate sample mean and covariance by unifying the samples of P and Q, which yields an upper bound of H P+2) and the bound is tight when = Q.
Putting those together, we reach our distance function for coarser scale objectives
det E
JSD(F. xe log det Eg det Ege
(4)
Another well-established metric between two Gaussian distributions is Bhattacharyya distance Dn: Df(f.f)) - (Ms "
1 log Vdct det Ey-
det Σ
(5)
2
Note that though JSD) and MCR in Dai et al. (2022) are constructed similarly, the latter computes X instead of the covariance matrix and is interpreted from coding rate reduction perspective.
X
4

---

Under review as a conference paper at ICLR 2023
E, Though having different geometric interpretations, it is notable that Dg and JSD have similar terms, including determinants of the covariance matrix. In the experiments, observed that these two distances yield similar performance in terms of learned representation, but JSD has a slightly faster convergence rate and better quality for generated images. Therefore, without further specification, we use JSD as our default choice for d((-).
where Σ
Fine-grained Optimization with Clustering. We perform Mean Shift clustering on f by grouping nearby samples. During grouping, we simplify the clustering process by equally averaging each neighbor sample rather than using feature similarity to reweight their contribution. To improve nearest neighbor search stability, we maintain a rolling updated memory bank fm as a querying pool and use the backbone representation f" rather than f as the key to computing feature similarity. Denoting {fig) as the returned K' nearest neighbors for fi, our clustering objective is:
N
K
Cgroup
max D NK
Note that we do not have repulsive term here since we empirically find Eqn.2 suffices to maintain feature diversity to prevent collapsed solution. Additionally, we can leverage the clustering objec- tive to perform adversarial optimization by alternatively differentiating and attracting nearby fake features f to embed l into the clusters eventually. Similarly, with (S 1 denoted as returned K nearest neighbor real embedding for fú, we define an adversarial grouping objective as:
Lady group min max
IC-GAN(Casanova et al., 2021) implements similar instance-wise objective. However, they use an off-the-shelf model rather than jointly optimized embedding search for the nearest neighbor. During training, we jointly optimize Leluster == Lgroup + Ladv group for clustering objectives.
3.3 SMOOTHING REGULARIZATION
Besides re-founulg adversarial targets for representation learning, we need to target another common issue in GAN's training: balancing the model's capacity and the smoothness constraint.
Recent studies demonstrate that regularizing D's smoothness, or its Lipschitz constant, is critical for scaling GAN to a giant architec- ture. Consider continuous function F ITM + IP. It is well known that we can bound its Lipschitz constant by the spectral norm of Jacobian Jp:
Algorithm 1: Approximating || | with
power iterations
Input: Function F Thm + RR; Stop gradient operator (g):); Power iteration steps s; Batch size b; Input data æ E 12xmm Initialize random vector u NO 1) R for iter ~ 1... do U [ u = (F
|| [ S Lip,
1/VJP
where ||-112 denotes the matrix spectral norm. We can efficiently approximate it by adopt- ing the power-iteration (Miyato et al., 2018).
1/JVP
end Return: ||J+(m))]=
sg =)())))(())))()))
However, computing the full Jacobian matrix is highly inefficient in the current deep learning package where they implement backpropagation us- (VJP) subroutine. Each backpropagation only computes a single row of the Jacobian matrix. In our case, it requires p times backpropagation call to compute full Jp, which is impracticable as we usually need substantial p to embed representation.
ing
Leveraging the fact that power-iteration is matrix-free method, we do not need to explicitly com- pute the Jacobian matrix. Instead, we only need to access the matrix by evaluating the matrix-vector product, which can be efficiently computed by batch-wise VJP and JVP (Jacobian- -ector--rooor))) subroutine. Algorithml presents the details, where only (2S + 1) times of back-propagation are required to approximate [IJ]( In our experiments, we find that S = 1 suffices for a ResNet-18 model.
We observe that regularizing (ω 12 normalized embedding f will enlarge the norm of f throughout training and eventually destabilizes the system; while regularizing Jn(x) on f operates
5

---

Under review as a conference paper at ICLR 2023
oppositely. In our method, we choose to compute (zx on f and add extra hinge regularization to sustain the embedding norm at a regular level. Therefore, our smoothness regularization is:
regg == min Em Jo (x) - Lipll2 + - || 11/2.
(6)
where Å denotes the ratio for hinge regularization, and Lip denotes the Lipschitz target of D, which set to by default. Unlike layer-wise normalization scheme, e.g., Spectral Norm(Miyato et al., 2018), where demanding local regularization hurts the model's capacity; proposed regulariza- tion scheme allows the network to simultancously fit for multiple objectiveness, ie., representation learning and smoothness regularization. Therefore, the model does not have sacrifice capac- ity for smoothness. Another benefit of our methods is that the proposed term can work with the normalization layer, while Spectral Norm can't because of the data-dependent scaling term in the normalization layer.
Overall Objective. Combining all the aforementioned terms, we define our final objective as:
C == LGaussian + >eoCluster + Xa Xehregy
where λe, A control the relative loss weights.
4 EXPERIMENTAL SETTINGS
Datasets. We conduct experiment on three benchmark datasets: CIFAR-IO, CIFAR-100 (Krizhevsky et al., 2009) and ImageNet-10. ImageNet-10: we follow (Chang et al., 2017) to se- lect 10 categories from ImageNet dataset (Deng et al., 2009) resulting 13,000 training images and 500 validation images. During training, we only perform spatial augmentation, including random spatial cropping and horizontal flipping, followed by resizing images to 128x 128 resolution to match the generated images. During testing, resized the images to align the smaller edge to 144, fol- lowed by central cropping to produce a 128x128 output. the same augmentation strategy as in ImageNet-10 10 but produce 32x32 images; During testing, do not perform cropping.
During training, we apply
For compared methods, we keep their default augmentation strategy. In ImageNet-10, 10, resized ther augmented images to 128x 128. We run an unsupervised learning algorithm for all methods on the training split and evaluate the validation split.
Model Details.
Diseriminator: We construct our discriminator using ResNet-18 (He et al., 2016) and perform several modifications to make it cooperate reasonably with the generator: Inspired by the dis- criminator configuration in BigGAN, we perform spatial reduction only within the residual block and replace all stride two convolution layers with avg pooling followed by stride one convolu- tion. We remove the first max pooling layer and switch the first convolution layer to 3x3 kernel with 1xl stride to keep the resolution unchanged before the residual block.
To maintain a substantial down sample rate in ImageNet-10 10 images, we duplicate the first resid- ual block and enable a spatial reduction in all blocks to reach a 32 down-sample rate. In DIFAR-10/100
preserve the default setting for residual blocks. As our proposed smooth- ness term regularizes on each sample, we replace all BatchNorm layer (loffe & Szegedy, 2015) with GroupNNormWWu & He, 2018), specifying 16 channels as a single group, to prevent batch- wise interaction. We also remove the first normalization layer in each block as it produces better results. For activation function, we replace ReLU with ELU (Clevert et al., 2015) for broader non-linear support on negative value.
Generator: We adopt the default generator configurations from BigGAN-deep(Broe et al., 2019). Specifically, we take their model for 32x32 images for the experiment in the CIFAR dataset. We additionally increase the base channel 128 to prevent image generation from be- ing the system bottleneck. For ImageNet-100 10, we directly take their settings for 128x 128 images.
Optimization Details. We train our model for 1000 epochs in CIFAR-10/100 and 500 epochs in ImageNet-10. We optimize our objectives using AdamW optimizer (Loshchilov & Hutter, 2017) and use constant learning rate 2c-4 for both generator and discriminator, We additionally add 0.1 weight decay to the discriminator. We use batch size 500 in CIFAR-10/100 and 320 in ImageNet- 10. We small-scale parameter tunning experiment for hyperparameters and find that setting

---

Under review as a conference paper at ICLR 2023
(a) Data samples (b) Samples from Gen- categorical assignment
(c) Visualize representa- tion with t-SNE
(d) Visualize representa- tion with 2d palette map
erator
We train GAN with objectives synthetic double spiral data to qualitatively demon- strate the generated samples and learned representation. (a) Data sample colored by two ground truth categories (b) Generated samples that capture all modes and accurately avoid invalid regions, the space between two double spirals. (c) We use t-SNE to project the learned representation into two- dimensional space and color each point using ground truth assignment. We show that the learned representation correctly separates two categories and preserves continuity within each group. (4) We further showcase the point-wise features by projecting the representation into a 2d palette map. displayed at the bottom right corner, using t-SNE, illustrating that the embedding precisely preserves the point-wise spatial order across and within each group. Best viewed color.
Figure 2:
Ab, Aer A to 4, 3, yields the best result. We run single discriminator updating before optimizing generator, i.e. 1. for simplicity.
In CIFARI0/100 experiment, we use default configurations from SOLO-Learns (da Costa et al., 2022), an open source library providing heavily tuned configurations for multiple state-of-the-art self-supervised methods. In ImageNet-10 experiments, we train the compared approaches using the suggested hyperparameters for ImageNet-100 but extend the total epochs 1000 for sufficient convergences. We run their methods with our modified backbone and resize input images to 128x 128 for a fair comparison.
As widely adopted trick in GAN's training, we maintain momentum updated discriminator and generator for evaluation purposes and find produces stable data representation and better image quality. We also try producing f from momentum models for nearest neighbor searching, which brings slightly better performance in all benchmarks.
RESULTS AND DISCUSSION
5.1 SYNTHETIC DATA
Firstly, demonstrate our learned representation by running objectives synthetic double spirals dataset (Li et al., 2022). In this experiment, we implement discriminator and generator as multi-layer perceptions and keep all other configurations, e.g., normalization layer, choices of acti- vation, objectives function, learning rate, consistent with our settings for real images experiments. In Fig. 2a, we demonstrate that the generated samples precisely capture all data modes, and substan- tially avoid the outlier samples between spirals. Besides the generation capability, we also visually inspect the learned representations using t-SNE (Van der Maaten Hinton, 2008) and color each point using ground truth assignment. As shown in Fig.2c 2c, the embedding from the two categories is substantially separated.
In Fig.2d, we further illustrate the detailed structures of the learned representation where we color each data point by projecting Iearned representation into a 2d palette map, (displayed at the bottom verify the learned embedding preserves relative data structures within
right). From the plot, and across groups
5.2 REPRESENTATION LEARNING ON REAL IMAGES
We task the backbone of the discriminator to produce vector data representation and then evaluate its performance on the image classification task. We compare the results with offte- art contrastive learning approaches under two widely adopted evaluation metrics:

---

Under review as a conference paper at ICLR 2023
Method
CIFARIO
CIFARIO0
ImageNet10
SVM
K-means
SVM
K-means
SVM
K-means
Supervised baseline
95.13
95.15 (0.00)
75,96
73.60 (0.78)
96,40
96.37 (0.15)
Random
42.96
22.08 (0.09)
18.39
8.99 (0.16)
48.20
28.34 (0.66)
NNCLR (Same View)
29.53
20.21(0.22)
7.70
7.25(0.07)
DINO
89.73
63.90 (0.64)
65.64
36.73 (0.47)
87.80
67.98(2.25)
NNCLR
91.69
69.32 (3.20)
69.77
40.42 (0.62)
91.40
66.83 (3.20)
SimCLR
90.64
75.25 (3.32)
65.61
41.36 (0.59)
89.00
65.73 (2.87)
BYOL
93.08
75.03 (2.74)
70.69
42.80 (0.71)
90.40
67.30 (3.34)
SWAV
89. 14
64.50 (0.91)
65.07
35.26 (0.48)
90.01
61.94 (3.11)
Ours
89.76
80.11 (2.84)
63.34
38.16 (0.45)
91.20
75.41 (2.14)
Table 1: We evaluate our methods by benchmarking the learned representation using linear SVM and K-means clustering. We report averaged accuracy and standard deviation (shown in paren- theses), over 20 runs. Our method achieves competitive performance with state-of-the-ar self- supervised methods across multiple datasets. However, do not leverage any augmented views. With NNCLR (Same View), we additionally show removing augmented view information from the NNCLR pipeline yielding degenerated model.
Linear Support Vector Machine(SVM): We optimize Linear SVM on top of the 12 normalized training feature and report the classification accuracy on the validation set
K-means clustering: We run spherical K-means clustering on the validation set with K equaling the number of ground truth categories. We then obtain a prediction on the validation set by solving the optimal transport problem between clustering partition and ground truth categories. To reduce the randomness clustering, we perform five consecutive runs and pick the best clustering result with minimal inertial, the the distance between samples and centroids. We then repeat this process 20 times and report the averaged result and standard deviation. Compared to Linear SVM, K-means clustering evaluates the representation
more rigorous setting.
We report the results in Table. and compare our approach with current ofte-off-hhearr methods. For datasets with fewer categories, i.e., CIFAR-10 and ImageNet-10 10, our method significantly out- performs the state-of-the-art contrastive learning approaches in K-means clustering. In CIFAR-10, we get 80.11 test accuracy defeating the hest-competed method SimCLR with 75.25 test accuracy; In ImageNet-10. our methods reaching 75.41 test accuracy surpasses the best-competed method, DINO with 67.98 test accuracy. When evaluating representation using linear SVM, our method reaches 89.76 76 test accuracy, which exceeds SWAV, and DINO with 89. 14 and 89.73 test accuracy, respectively, but slightly worsens than B BYOL with 93.08 and NNCLR with 91.69 test accuracy. In ImageNet10, our method gets 91.20 which approaches the best method NNCLR with 91.40 and exceeds the rest.
In CIFAR-100, there are fewer training samples per category and operationalizing instance-wise discriminating objectives is thus favorable over clustering objectives or smoothness regularization. linear SVM metric where we get 63.34 ac-
Under such case, our methods remain competitive curacy, which is very close to DINO, SimCLR, and SWAV, which have around 65 test accuracy. On K-means clustering, our method reaches 38. 16 test accuracy outperforming clustering-basee con- trastive approaches SWAV and DINO, which have 36.73 and 35.26, respectively. In both metrics, our methods are behind the BYOL and NNCLR. We want to point out that, as the value of unsuper- vised training is on scaling with more data, evaluating self-supervised methods under constrained datasets, in CIFAR-100 100, is less informative than CIFAR-10 and ImageNet-
The quantitative comparison can be further qualitatively confirmed by visualizing embedding using (-SNE, BYOL, as shown in Fig. 3a, produces isolated and smaller-sized clusters that maintain sufficient space to discern categories under linear transformation. However, those clusters lack sufficient global-wise organizations, which is evaluated with K-means clustering. On the contrary, approach, shown in Fig. 3b and 3c, produces smoother embedding, which are nearly aligned with the ground truth partition yielding good performance on K-means clustering. However, our method's cluster has more outliers, leading slightly worse linear separability, measured in
8

---

Under review as a conference paper at ICLR 2023
(a) BYOL
(b) Full Objectives
(c) LGaussia only
(d) Hinge Loss
Figure 3:
On the CIFAR-10 validation set, we visualize the learned representation using t-SNE and compare with multiple baseline objectives. Training GAN with Lasssisn only produces diversified yet over smoothed representation. With cluster loss, the representations from full objectives ex- periment enhance categorical-wic discrepancy which are visually competitive with state-of-the-ar BYO..
Method
Inception Scores f
FID +
KMeans
SVM
BigGAN(Brock et al., 2019)
8.22
17.50
29.69
69.31
Hinge Loss
8.13
18.54
36.41
77.19
Gaussian only, with DB
8.39
17.83
70.76
87.9
Gasssiaa only, with JSD
8.55
16.97
80.55
88.32
Full objectives
8.73
13.63
80.11
89.76
Table 2: We run ablated experiments and compare different objectives function. Our proposed structural objectives significantly outperform widely adopted hinge loss baseline by large margin. Including finer scale clustering objectives (last row) improve both representation and image quality over the methods only using coarse scale objectives (3rd and 4th row).
linear SVM. Adding cluster loss will increase the categorical- discrimination. Moreover, yield better SVM results.
We run additional baseline experiment by removing the view augmentation from NNCLR pipeline, termed as NNCLR(same view) in 1. In this variant, we only align the feature with the nearest neighbor from the memory bank and repulse all other features. Interestingly, such modification breaks the model in all settings and leading even worse results than the randomly initialized baseline. We hypothesize that, without an augmented view providing extra information, nearest neighbor searching does not return useful features, and the model is mostly likely over-fitting noisy artifacts during training.
5.3 IMAGE QUALITY
As shown in Table 2, provide a quantitative comparison across multiple methods. We show that the proposed objectives significantly improve generation and representation quality over the hinge loss baseline. We can witness further enhancement in image quality with clustering loss that provides extra instance/clustering-wi surpasses our hinge loss baseline since BigGAN has a smaller batch size resulting more updating rounds in the momentum Generator. We provide samples of generated images in the Appendix.
objectives. Note that the unconditional BigGAN baseline
CONCLUSION
We propose structural adversarial objectives that augment the GAN framework for self-supervised representation learning. The objectives shape the discriminator's output at two granularity levels: at coarse scale, features are optimized with Gaussian assumption yielding diversified and smooth rep- resentation, while at a finer scale, the nearby features are grouped, forming local clusters. We bench- mark across multiple datasets and show supplementing GAN with these self sulf-supe objectives suffices to produce data representation that is competitive with the state-of- -f-the- self-supervised learning approaches and substantially improves the quality of generated images.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
Hangbo Bao, Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint N0Niv:2066.082444 2021.
Vineeth S Bhaskara, Tristan Aumentado-Ammste
Allan D Jepson, and Alex Levinshtein. Gran- gan: Piecewise gradient normalization for generative adversarial networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3821-3830, 2022.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL
Sttts:://ppereee
net/ / forum 1id
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9922,, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-super vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Arantxa Casanova, Marlene Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero Soriano.
iustance-conditione gan. Advances in Neural Information Processing Systems, 34:27517-27555 2021.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adap- tive image clustering. Proceedings of the IEEE international conference on computer vision, pp. 5879-587789 2017.
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In Proceedings of the IEEE international conference on computer vision, pp. 1511-1520, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Djork-Aré Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07288 2015.
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo- learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 23(56): 6, 2022. URL http: jmlr. org/ wwwpg//pgg///////////t//////eeee paaprs/v23/21-111111 html.
Xili Dai, Shengbang Tong. Mingyang Li, Ziyang Wu, Kwan Ho Ryan Chan, Pengyuan Zhai, Yaodong Yu, Michael Psenka, Xiaojun Yuan, Heung-Yenn Yeung Shum, and Yi Ma. Closed-loop data transcription to LDR via minimaxing rate reduction, 2022. URL https: /ttp/////peeee...e/e/ net net/forum?? stt.:mmmoids5111ssru030333
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255 2009. doi: 10. 10..1099.PP.2209990. 52066484
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In CVPR, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances in neural information processing systems, 32, 2019.
10

---

Under review as a conference paper at ICLR 2023
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In International Conference on Learning Representations, 2017. URLhttps: BoLLttpss:/openzeiiiie formm?
id-BJtNZAFgg.
Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems, 34:15608- 15620, 2021.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martin Arjovsky, Olivier Mastropi- etro, and Aaron Courville. Adversarially learned inference. In International Conference Learn- ing Representations, 2017. URL
net forum deidiid-BBlBee 1
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9588-9597, 2021.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, (3(11) 1):199144, 2020.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 2020.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 973899778.110 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders scalable vision learners. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16000-16009) 2022.
John R Hershey and Peder A Olsen. Approximating the kullback leibler divergence between gaus- sian mixture models. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSPP volume 4, pp. IV-317. IEEE, 2007.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Semantic object accuracy for generative text-to- image synthesis. JEEE transactions on pattern analysis and machine intelligence, 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 10/96, 2017."
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410 2019.

---

Under review as a conference paper at ICLR 2023
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110-8119 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Process- ing Systems, 34:852-863, 2021.
Diederik Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. In CVPR, 2017.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. Advances in neural information processing systems, 30, 2017.
Zengyi Li, Yubei Chen, Yann LeCun, and Friedrich T Sommer. Neural manifold clustering and embedding. arXiv preprint arXiv:2201.10000 2022.
Jae Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint arXiv: 2017
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Xuezhe Ma, Xiang Kong, Shanghang Zhang, and Eduard H Hovy. Decoupling global and local representations via invertible generative flows. In International Conference on Learning Repre- sentations, 2021. URL https: /ttps:://ppeeeee /penrevieeeee n forum? 1d=1WLBy
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint
2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv: 10X7vv:807..0377 007700777.... 2018.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2536 2536-2544, 2544, 2016.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakom. Dif- fusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10619-10629, 2022.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.0644 2015.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-6839, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Feng Wang, Tao Kong, Rufeng Zhang, Huaping Liu. and Hang Li. sulf-supe learning by estimating twin class distributions. arXiv preprint arXiv:2 10.07420 2021.
12

---

Under review as a conference paper at ICLR 2023
Yi-Lun Wu, Hong-Han Shuai, Zhi-Rui Tam, and Hong- Yu Chiu. Gradient normalization for gener- ative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pp. 6373 0373-6382, 2021.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. International Conference on Machine Learning. PMLR, 2021.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang, Cross-modal contrastive learning for text-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 833 842, 2021.
Mingtian Zhang, Tim Z. Xiao, Brooks Paige, and David Barber. Improving vae-based representation learning, arXiv preprint arXiv:2205.14539 2022.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
Xiao Zhang and Michael Maire. Self-supervised visual representation learning from hierarchical grouping. Advances in Neural Information Processing Systems, 33:16579-16590, 2020.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference computer vision, pp. 2223-2232, 2017.
A APPENDIX
Compared Methods To evaluate the learned representation, we compare our methods with state- of-the-art art self-supervised learning methods: SimCLR(Chen al., 2020) optimizes the InfoNCE loss, maximizing feature similarity across views while repulsing all the images. NNCLR(DWibed. al., 2021) sample nearest neighbour from data set using cross view features and treat them as positive for InfoNCE objectives. We additionally run a baseline using NNCLR, removing the aug- mented view and directly maximizing the similarity between image features and its nearest neighbor. DINO(Caron et al., 2021) and SWAV(Caron et al., 2020) maximizing view consistent objectives us- ing clustering- based targets. YOL(Grril et al., 2020) only contains the maximizing term and adopts a momentum updated Siamese model to process augmented input to prevent collapsed solution.
For image generation, we compare with the BigGAN(Brock al., 2019), showing the superior capability to generate complex images. For a fair comparison, we run BigGAN in unconditional mode.
13

---

Under review as a conference paper at ICLR 2023
Figure 4: Randomly generated images from our full objectives.
14

---

Under review as a conference paper at ICLR 2023
Figure 5: Randomly generated images from GAN trained with LGaussian only
15

---

Under review as a conference paper at ICLR 2023
Figure 6: Randomly generated images from GAN trained with hinge loss
16

---

Under review as a conference paper at ICLR 2023
Figure 7: Randomly generated images from unconditional BigGAN
17


Reviews:
[{"review_id": "_OW2j0ohJIY", "reviewer": "Reviewer_XWaT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a GAN variant with structural adversarial objectives for self-supervised (SS) representation learning, which aims to achieve a general SS representation learning, especially escaping dependence upon the hand-crafted elements guiding data augmentation or proxy task design. In particular, at a coarse scale, a JSD divergence between the discriminator features of real samples and generated samples is defined as an adversarial objective for GAN.  At a finer scale, an adversarial clustering objective is defined, which groups adjacent real embeddings (features extracted by the discriminator) to form a cluster and adversarially attracts the generated embedding towards the nearby cluster center. Furthermore, an efficient regularization scheme that approximates the spectral norm of the Jacobian is introduced to regularize the discriminator’s smoothness. Experiments demonstrate their proposed GAN achieves results that compete with networks trained by state-of-the-art contrastive approaches.", "summary_of_the_review": "The proposed structural adversarial objectives are interesting and seem novel. The performance of the proposed GAN especially on K-mean clustering is promising. However, the authors do not well position their work and miss the comparison with closely related works. Thus, the contributions of this work cannot be clearly demonstrated. In addition, the empirical support is not consistent with the original motivation. Last, the clarity of this work needs improvement.", "strength_and_weaknesses": "The proposed structural adversarial objectives are interesting and seem novel. The performance of the proposed GAN especially on K-mean clustering is promising. However, I have the following concerns:\n1. Actually, this is a kind of work that utilizes a generative model together with clustering for SS representation learning. Therefore, deep generative clustering, a.k.a., generative model (GAN/VAE) with clustering, e.g., [1,2,3,4], is closely related to this work, which is missed in the Related Work part of this paper.\n2. I think the proposed work can be better positioned in the literature if the motivation is to target deep generative clustering instead of SS representation learning. I would expect a detailed analysis about the superior performance of the proposed generative clustering to the existing generative clustering works.\n3. [1] also uses discriminator features for clustering. From this aspect, the idea claimed in this work that tasks the discriminator with additional structural modeling is not novel.  \n4. The experimental study is not totally consistent with the original motivation. First, augmentation is adopted though simple. Second, the experiments can be more convincing if conducted on different modalities of data, like text. It can verify the generality of the proposed method and also demonstrate the data augmentation should be specified in terms of data domains.\n5. Gaussian assumption encouraging smoothness and diversified representation is not well justified.\n \n\nSome questions:\n1. Why spectral normalization used in Miyato et al. 2018 will harm model capacity but the spectral normalization used in this work will not?\n2. How to decide which layers of D as clustering features?\n3. What is the size of the maintained memory bank $f_m$?\n4. For coarse-scale optimization with Gaussian, is it using mini-batches to determine the covariance and mean? If so, is it problematic since Eq. (4)/(5) is supposed to be defined on the whole data?\n5. Why “regularizing JD(x) on l2 normalized embedding f will enlarge the norm of f˜ throughout training and eventually destabilizes the system; while regularizing JD(x) on f˜operates oppositely”?\n\n\nErrors\n1. In Section 2.2, “and demonstrates their sensitivity to the parameters of augmentation schemes.”-→demonstrate\n2. In Eq. (2), JSD should equal to $1/2*D_{KL}$…\n3. In $L_{advgroup}$, $f_i^g$ is supposed to be $f_i$?\n4. $\\\\{f_{i,j}\\\\}_{j=1}^{k}$, $k$ is supposed to be $K$.\n5. In Alg 1, $s$ should be $S$.\n6. Fig. 2, (4) should be (d).\n\nReferences\n\n[1] Liu, S., Wang, T., Bau, D., Zhu, J. Y., & Torralba, A. (2020). Diverse image generation via self-conditioned gans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14286-14295).\n\n[2] Noroozi, M. (2020). Self-labeled conditional gans. arXiv preprint arXiv:2012.02162.\n\n[3] Guo, X., Gao, L., Liu, X., & Yin, J. (2017, August). Improved deep embedded clustering with local structure preservation. In Ijcai (pp. 1753-1759).\n\n[4] Jiang, Z., Zheng, Y., Tan, H., Tang, B., & Zhou, H. (2017, January). Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering. In IJCAI.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RZ8nUqPu_5", "reviewer": "Reviewer_9r54", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to use the discriminator in a GAN as a feature extractor for self-supervised representation learning. Assuming that both real and fake features from the discriminator follow a gaussian distribution, the authors propose a loss based on the distance between the real and fake gaussian distributions. They also propose performing clustering on the extracted features and use this as an extra loss term for the discriminator optimization. Their proposed method outperforms SOTA self-supervised learning approaches on small-scale datasets and using the K-means metric.", "summary_of_the_review": "The paper has both strengths and weaknesses. I have some concerns regarding the evaluation setting and discussion on existing literature.", "strength_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to follow.\n2. The paper is well-motivated, and it is technically and theoretically sound.\n3. The proposed model outperforms BigGAN in terms of image generation quality.\n4. The idea of self-supervised feature representation learning using a discriminator is interesting and novel to the best of my knowledge.\n\nWeaknesses:\n1. There are some grammatical errors in the text, and some sentences are too long to follow. E.g.: Page 1, Yet, to improve...\n2. There is no discussion on different discriminator architectures, such as the patchGAN discriminator or feature extractors as discriminators [1,2]. Also, a more detailed comparison against ICGAN can improve the paper.\n\n[1] Sungatullina, Diana, et al. \"Image manipulation with perceptual discriminators.\" ECCV 2018.\n\n[2] Mao, Xin, et al. \"Is discriminator a good feature extractor?.\" arXiv 2019.\n\n3. The evaluation setting is not common in self-supervised approaches. Usually, linear classification and kNN classifiers are used. In this paper, they use SVM and Kmeans.\n\n4. The computational complexities of the networks in Tab. 1 are not compared.\n\n5. The proposed model only outperforms previous work on one metric.\n\nMinor: \nThe references for related work in Tab. 1 could be added.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1DxISdhCmD", "reviewer": "Reviewer_GyrJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the problem of representation learning with GANs. To do so, it introduces a novel adversarial objective that is composed of a distribution-matching objective and a clustering objective. In addition, they propose a novel regularization algorithm to ensure the smoothness of the discriminator. Compared to the widespread spectral normalization, the proposed regularization does not control the spectral norm at each layer individually, allowing the discriminator to retain more capacity. The method is evaluated against contrastive state-of-the-art, on CIFAR-10, CIFAR-100, and ImageNet-10.", "summary_of_the_review": "The work has relevant contributions for both GANs in general as well as their application to representation learning.\nIt tackles the problem in a novel and interesting way.\nHowever, too many details are missing, as well as comparisons, and some related work.", "strength_and_weaknesses": "Strengths:\n- Representation learning is a timely topic. Tackling it with generative models has been a promising approach but their usefulness had yet to be proven compared to their contrastive counterparts. The current submission is a big step in that direction.\n- The approach is original in significant ways. It focuses on the discriminator, which is not usually exploited in GAN literature, and manages to use it as a feature learner.\n- It also proposes a new GAN loss and a new GAN regularization scheme, which both seem useful for training GANs.\n\nWeaknesses:\n- The presentation of the technical parts could be made clearer:\n  - In section 3.2, it seems that $f$ represents alternatively the individual outputs of $D$ as well as their distribution. It could be useful to clarify the notation. It can be especially confusing since the original adversarial objective, i.e. the binary classification objective between fake and real samples, can sometimes be presented as a JSD minimization of the distributions.\n  - It could be useful to remind the reader of how VJP and JVP can be computed in the supplementary. I for one would welcome it.\n  - Table 2 contains important information as it serves as an ablation study regarding the capacity of GAN discriminators to learn features. I don't think this aspect is discussed anywhere in the paper. Also, it isn't mentioned how the representations for the baselines in Table 2 have been extracted.\n  - Have the contrastive baselines been trained on the datasets they are being evaluated on, like the GAN would be? Or have they been trained on ImageNet and evaluated only on those datasets, as is often the case in contrastive benchmarks? This information is important to put the results in context and should be made explicit.\n- The paper should compare to other generative representation learning methods such as those they cite in their related work, or a more recent one [1].\n- The authors might also want to discuss how their objectives relate to and differ from Feature Matching [2]. Here are some more recent examples of where it has been used [3,4].\n- The BigGAN backbone is quite generic and useful, but somewhat outdated, especially for unconditional generation. Have the authors considered StyleGAN-2 [5] or FastGAN [6] backbones? Can the author discuss the potential issues of switching to such backbones?\n\n---\n\n[1] Ali Jahanian, Xavier Puig, Yonglong Tian, Phillip Isola. Generative Models as a Data Source for Multiview Representation Learning. ICLR 2022.\n\n[2] Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen. Improved Techniques for Training GANs. NeurIPS 2016.\n\n[3] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, Bryan Catanzaro. High-Resolution Image Synthesis and Semantic Manipulation With Conditional GANs. CVPR 2018.\n\n[4] Liming Jiang, Changxu Zhang, Mingyang Huang, Chunxiao Liu, Jianping Shi, Chen Change Loy. TSIT: A Simple and Versatile Framework for Image-to-Image Translation. ECCV 2020.\n\n[5] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, Timo Aila. Analyzing and Improving the Image Quality of StyleGAN. CVPR 2020.\n\n[6] Bingchen Liu, Yizhe Zhu, Kunpeng Song, Ahmed Elgammal. Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis. ICLR 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject