Distilling Representations from GAN Generator via Squeeze and Span
Authors: Yu Yang
Xiaotian Cheng
Chang Liu
Hakan Bilen
Xiangyang Ji
Venue: NeurIPS 2022 Accept (2022)
Keywords: Representation Learning
Generative Adversarial Network
Knowledge Distillation
Self-Supervised Learning

Abstract: In recent years, generative adversarial networks (GANs) have been an actively studied topic and shown to successfully produce high-quality realistic images in various domains. The controllable synthesis ability of GAN generators suggests that they maintain informative, disentangled, and explainable image representations, but leveraging and transferring their representations to downstream tasks is largely unexplored. In this paper, we propose to distill knowledge from GAN generators by squeezing and spanning their representations. We \emph{squeeze} the generator features into representations that are invariant to semantic-preserving transformations through a network before they are distilled into the student network. We \emph{span} the distilled representation of the synthetic domain to the real domain by also using real training data to remedy the mode collapse of GANs and boost the student network performance in a real domain. Experiments justify the efficacy of our method and reveal its great significance in self-supervised representation learning. Code is available at https://github.com/yangyu12/squeeze-and-span.

Distilling Representations from GAN Generator via Squeeze and Span
Yu Yang!", Xiaotian Cheng', Chang Liu', Hakan Bilen', Xiangyang Tsinghua University, BNRist Unnierrsity of Ediburgh yang yul6@foxmail,..... com, cxt200mails 2 edu on (1iuchang2022, xyjil@tsinghua. edu. cn, hbilen@ed. uk
Abstract
In recent years, generative adversarial networks (GANs) have been actively stud- ied topic and shown to successfully produce high-quality realistic images various domains. The controllable synthesis ability of GAN generators suggests that they maintain informative, disentangled, and explainable image representations, but leveraging and transferring their representations to downstream tasks is largely unexplored. In this paper, we propose to distill knowledge from GAN generators by squeezing and spanning their representations. We squeeze the generator fea- tures into representations that are invariant to semantic-preserving transformations through network before they are distilled into the student network. We span the distilled representation of the synthetic domain to the real domain by also using real training data to remedy the mode collapse of GANs and boost the student network performance in real domain. Experiments justify the efficacy of our method and reveal its great significance in self supervised representation learning. Code is available at https: / /etee://gitttee github. www/ yaamm//angy12//// yaaggyui//squuueeeeeeeeee squeeze- and- span.
Introduction
Generative adversarial networks (GANs) [23] continue achieve impressive image synthesis results thanks to large datasets and recent advances in network architecture design [5, 36, 37, 34]. GANs synthesize not only realistic images but also steerable ones towards specific content or styles [22, 52, 49,33, 57, 53,32), 32). These properties motivate a rich body of works to adopt powerful pretrained GANs for various computer vision tasks, including part segmentation [68, 56, 61]. 3D reconstruction [67], image alignment [48, 45], showing the strengths of GANs in the few-label regime.
GANs typically produce fine-grained, disentangled, and explainable representations, which allow for higher data efficiency and better generalization [42, 68, 56, 61, 67, 48]. Prior works on GAN-based representation learning focus on learned features from either a discriminator network [50] or an encoder network mapping images back into the latent space [19, 17, 18]. However, there is still inadequate exploration about how to leverage or transfer the learned representations in generators. Inspired by the recent success of [68, 56, 61], we hypothesize that representations produced in generator networks are rich and informative for downstream discriminative tasks. Hence, this paper proposes to distill representations from feature maps of pretrained generator network into student network (see Fig. l).
In particular, we present a novel "squeeze-and-span" technique to distill knowledge from a generator into a representation network2 that transferred downstream task. Unlike transferring discrimi-
"qqual Contribution
Throoughout the paper, two terms" renreseepestntttiionn network" and "student network" are used interchangeably, are the "generator network" and "teacher network".
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Legend
space
Supervision
Transferalle representation
w P(w)-
w
P-((W)
W P(w
fake
Xrec
G
X
Fake
D
Xyeal
E
S
(a) Discrimination of real and fake
(b) Encoding into latent space
(c) Distillation generator feature
Figure I: A comparison of representation transfer in GANs. (a) Transferring representations in
discriminator (D) which is tasked distinguish real fake images (e.g. [50]). (b) Transferring
representations in encoder (E) which projects an image into latent space (e.g. [19, 17, 18]). (c) Transferring representations in student (S) which predicts the generator features (ours).
nator network, generator network is not directly transferable to downstream image recognition tasks, as it cannot ingest image input but a latent vector. Hence, distill generator network representa- tions into a representation network that can be further transferred to the target task. When fed in a synthesized image, the representation network is optimized to produce similar representations the generator network's. s. However, the generator representations are very nigh-dimensional and not all of them are informative for the downstream task. Thus, we propose squeeze module that purifies generator representations to be invariant to semantic-preserving transformations through an MLP and augmentation strategy. As the joint optimization of the squeeze module and representation network can lead to a trivial solution (e.g. mapping representations to zero vector), we employ variance-covarraacee regularization in [3] while maximizing the agreement between the two networks. Finally, to address the potential domain gap between synthetic and real images, we span the learned representation of synthetic images by training the representation network additionally on real images.
We evaluate our distilled representations on CIFAR1O, CIFARIO0 and STL10 with linear classification tasks commonly done representation learning. Experimental results show that squeezing and spanning generator representations outperforms methods that build on discriminator and encoding images into latent space. Moreover, our method achieves better results than discriminative SSL. algorithms, including SimSiam [10] and VICReg [3] on CIFAR 10 and CIFAR 100, 100, and competitive results STL.10, showing significant potential for transferable representation learning.
We (I) provide new taxonomy of representation and transfer learning in generative adversarial networks based on the location of the representations, (2) propose a novel "squeeze-and--sp"" and transfer them for downstream tasks, (3) empirically show the promise of utilizing generator features to benefit self-supervised representation learning.
Our contributions can be summarized as follows:
framework to distill representations the GAN generator
Related Work
GANs for Representation Learning. Significant progress has been made the interpretability. manipulability, and versatility of the latent space and representation of GANs [36, 37, 35J. inspires a broad spectrum of GAN-based applications, such as semantic segmentation [68, 56, 61], visual alignment [48, 45), and 3D reconstruction [67], where GAN representations are leveraged to synthesize supervision signals efficiently. As GAN can be trained unsupervised. its representations are transferred to downstream tasks. DCGAN [50] proposes a convolutional GAN and uses the pre-trained discriminator for image classification. BiGAN [17] adopts an inverse mapping strategy to transfer the real domain knowledge for representation learning. While ALI [19] improves this idea with a stochastic network instead deterministic one, BigBiGAN [18] extends BiGAN with BigGAN [5] for large scale representation learning. GHFeat [59] trains post hoc encoder that maps given images back into style codes of style-based GANs [36, 37, 35] for image representation. These works leverage transfer representations from either discriminators or encoders. In contrast, our method reveals that the generator of pre-trained GAN typically more suitable for representation transfer with proper distillation strategy.

---

Deer
Fen
Baas
Skp
(a) Discriminator feature
(b) Latent variable
(c) Generator feature
Figure 2: Visualization of three types of GAN representations: (a) discriminator feature, (b) latent variable, and (c) generator feature. An unconditional StyleGAN2-ADA model pre-trained on CI- FARIO employed. Colors indicate different classes. Generator features are naturally clustered and consistent with classes.
Knowledge Distillation (KD) aims at training a small student network, under the supervision of relatively large teacher network [31]. In terms of the knowledge source, it can be broadly divided into logit-based KD and feature- -ased KD. Logit-based KD methods [41, 60, 12] optimize the divergence loss between the predicted class distributions, usually called logits soft labels, of the teacher and student network. Feature-based KD methods [38, 2, 54] adopt the teacher model's intermediate layers as supervisory signals for the student. FitNet [51] introduces the output of hidden layers of the teacher network as supervision. AT [63] proposes to match attention maps between the teacher and student. FSP [62] calculates flow between layers guidance for distillation. Likewise, our method distills knowledge from intermediate layers from a pre-trained GAN generator.
supervieer Representation Learning (SSL) pursues learning general transferable representa- tions from unlabelled data. To produce informative self-supervision signals, the design handcrafted pretext tasks has flourished for long time, including jigsaw puzzle completion [46], relative position prediction [15, 16), rotation perception [21], inpainting [47], colorization [40, 65], masked image intra-instance instance prediction, contrastive learning- based SSL methods explore inter-instane relation. Applying the InfoNCE loss or its variants [26], they typically partition informative positive/negative data subsets and attempt to attract positive pairs while repelling negative ones. MoCo series [28, 8, 11] introduce an offline memory bank to store large negative samples for contrast and momentum encoder to make them consistent. SimCLR [7] adopts an end-to-end manner to provide negatives in a mini-batch and introduce substantial data augmentation and a projection head improve the performance significantly. Surprisingly, without negative pairs, BYOL [25] proposes simple asymmetry SSL framework with the momentum branch applying the stop gradient to avoid model collapse. It inspires series of in-deep explorations, such as SimSiam [9]. Barlow Twins [64], VICReg [3], etc. In this paper, despite the same end goal of obtaining transferable representations and the use of techniques from VICReg [3], we study the trans- ferability of generator representations in pretrained GANs to discriminative tasks, use asymmetric instead of siamese networks, and design effective distillation strategies.
modeling [27, 588, etc. Instead of performing intra-
3 Rethinking GAN Representations
Let G W X denote generator network that maps a latent variable in W to image in X. An unconditional GAN trains G adversarially against a discriminator network D X + +00,1 that estimates the realness of the given images,
max min log(1 1 D(G(w))) + log D(x).
(I)
G
The adversarial learning does not require any human supervision and therefore allows for learning representations in an unsupervised way. In this paper, we show that the type of GAN representations and how they are obtained has a large effect on their transferrability. To illustrate the impact on the transfcrability, Fig. 2 plots the embedded 2D points of three different type of representations from an
3

---

unconditional GAN, where color is assigned based on the class labels.' Note that we describe each representation in the following paragraphs.
Discriminator Feature The discriminator D, which is tasked to distinguish real and fake images, can be transferred to various recognition tasks [50]. Formally, let D = d o d( o .... o d()) denote the decomposition of discriminator into L consecutive layers. As shown in Fig. 1(a), given an image x. the discriminator representation can be extracted by concatenating the features after average pooling from each discriminator block output,
h" ju(bi)..... (b()), ((b)))
where h = ad o... o........ dd
(2)
where ja denotes the average pooling operator. However, Fig. 2(a) shows that the cluster of dis- criminator features is not significantly correlated with class information indicating that real/fake discrimination does not necessarily relate to class separation.
Latent Variable An alternative way of transferring GAN representation is through its latent variable w [19, 17, 18J. In particular, one can invert the generator such that it can extract a latent variable representation of the generated image through learned encoder E. Then the representations of the encoder can be transferred to downstream task. While some works jointly trains the encoder with the generator and discriminator [19, 17, 18], we consider training a post hoc encoder [6] given fixed pre-traine pre-trained generator G, as this provides more consistent comparison with the other two strategies:
E" = arg min Bw (iW.--((w...ww [[G(((()) - x||1 + (G(E(x)),, + A| - w 1
E
where Epercep denotes the LPIPS loss [66] and X 1.0 is used balance different loss terms. The key assumption behind this strategy is that latent variables encode various characteristics of generated images (e.g. [33, 57J) and hence extracting them from generated images result in learning transferrable representations.
Fig, 2(b) visualizes the embedding of latent variables*. It shows that samples from the same classes are not clustered together and distant from other ones. In other words, latent variables do not disentangle the class information while encoding other information about image synthesis.
Generator Feature An overlooked practice to utilize generator features. Typically, GAN genera- tors transform low-resolution (e.g. 4x feature map higher-resolution one (e.g. 256x256) and further synthesize images from the final feature map [17, 36] or multi-scale feature maps [37]. The image synthesis is performed hierarchically: feature map from low high resolution encodes the low-frequency to high- nigh-frequency component for composing an image signal [35]. This understanding also evidenced by image editing works [22, 52, 49, 53, 32] which show that interfering with low-resolution feature maps leads to a structural and high-level change of an image, and altering high-resolution feature maps only induces subtle appearance changes. Therefore, generator features contain valuable hierarchical knowledge about image. Formally, let G g(z) o g((--)) ..... g())
denote the decomposition of a discriminator into L consecutive layers. Given a latent variable P(w) drawn from prior distribution, we consider the concatenated features average pooled from each generator block output,
h (p(h)). (h))), where h. ()) .... (w)
(4)
As Fig. 2(c) shows, generator features within the same class are naturally clustered. This result suggests that generators contain identifiable representations that can be transferred for downstream tasks. However, as GANs do not initially provide reverse model for the accurate recovery of generator features, it is still inconvenient to extract generator features for any given image. This limitation motivates to distill the valuable features from GAN generators.
'The GAN trained trained on CI We use UMAP embeddings [44] for dimensionality reduction. As the class labels of the generated images are unknown, they are inferred by a classifier that is trained on CIFAR10 training set and achieves around 95% top-1 accuracy on CIFARIO 10 validation set.
*In the family StyleGAN, to achieve more disentangled latent variables. the prior latent variable that observes standard normal distribution is mapped into learnable latent space via an MLP before fed into the generator. In our work, we refer to latent variables as the transformed ones, which are also known as latent variables in W space in other works [1]

---

Teacher
Teacher
Couse
Synthesis
w-P(w)
P
DoRGB
w- P(M)
Synthesis
a (x₂)
T
heyst & keow
Squeeze
DORGB
Lxor &
Synthesis
Fdi
(x,,)
S
P.
Luar fror
Genmaaee
IoRGB
Syuthesis
Student
LiD
Span
32x32
a' (x,)
Liur & feor
Squeeno
modile F..
IRRGB
1-0
Figure 3: Squceze and span representation from the GAN generator. Left: pretrained generator G and squeeze module T& constitute teacher network to produce squeezed representations which are further distilled into student network Sg (Squeeze part). The student network is also trained on real data (Span part). Right: the generator structure and squeeze module. We average pool (denoted as (1) the feature maps from each synthesis block and transform them with linear layer plus an MLP, termed squeeze module.
Squeeze-and-Span Representations from GAN Generator
This section introduces the "Squeeze-and-Sppe technique distill representation from GANs into student network, which can then be readily transferred for downstream tasks, egg. image classification. Let Sg H denote student network that maps given image into representation space. A naive way of representation learning can be achieved by tasking the student network to predict the teacher representation, which can be formulated the following optimization problem:
min P(w) [Sp(G(()) (W)) he (w)
(5)
where we use mean squared error to measure the prediction loss and (w ) to denote the dependence of h2 on w. However, this formulation has two problems. First, representations extracted through multiple layers of the generator are likely to contain significantly redundant information for down- stream tasks necessary for image synthesis. Second, as the student network only optimized on synthetic images, it is likely perform poorly in extracting features from real images in the downstream task due to the potential domain gap between real and synthetic images. To mitigate these issues, we propose the "Squeeze and Span" technique as illustrated in Fig. 3.
4.1 Squeezing Informative Representations
To alleviate the first issue that generator representation may contain a big portion of irrelevant information for downstream tasks, introduce squeeze (or bottleneck) module Tę (Fig. 3) that squeezes informative representations out of the generator representation. In addition, we transform the generated image via semantic-preserveee image transformation (e.g. color jittering and cropping) before feeding it to the student work. Equ. 5 can be rewritten as
min LRD Fww || 8,6
[|pp(a[((w)) - T (h
(6)
where image transformation a is randomly sampled from A. In words, we seek to distill compact representations from the generator among the ones that invariant to data augmentation A, inspired from the success of recent sulf-super methods [7, 10]. An informal intepretation is that, similar of the alternate subproblems that fix 0 and solve á, the optimal
to Chen & He [10], considering solution would result in the effect of Tg. (h)( = Ea-A (owa(G(w)). which implies Equ encourages Tę squeeze
transformation-ine -nviaiariiiinnniaantttt representation. However, similar to the siamese network in SSL [10], there exists a trivial solution to Equ. both the squeeze module and the student network degenerate output constant for any input.
Therefore, we consult the techniques from SSL methods and add regularization terms the distillation loss. In particular, employ variance-covariance [3] to explicitly regularize representations to be
5

---

significantly uncorrelated and varied in each dimension. Formally, in mini batch of N samples, we denote the squeezed generator representations and student representations with
2 = (T& ( (w2)), (w E MRN, Z, = (5=, where W; P(w) and a A denote random sample of latent variable and data augmentation operator. The variance loss is introduced to encourage the standard deviation of each representation dimension to be greater than 1,
(7)
So (G S Sa [G( E RMxx,,
(8)
L (Z)
max V Var(z>) + e);
M /=1
where 2 represents the j-th dimension in representation z. The covariance loss is introduced to encourage the correlation of any pair of dimensions to be uncorrelated,
Loov (7) मे7 [0
(10)
where C(Z) = N Σ (z: 2) 2)
2 = N 2 Z3
To this end, the loss function of squeezing representations from the generator into the student network can be summarized as
Lsqueze ALRD + μr [Cvrr (Z)) +
+ + (Zz) + Loow (Zg)]
(11)
Discussion Our work differs from multi-view representation learning methods [3, 10] in the follow- ing aspects. (1) Our work studies the transfer of the generative model that does not originally favor representation extraction, whereas most multi-view representation learning learns representation with discriminative pretext tasks. (2) Unlike typical Siamese networks in multi-view representation learning, the two networks in our work are asymmetric: one takes in noise and outputs an image and the other works in the reverse fashion. (3) While most multi-view representation learning methods learn representation networks from scratch, our work distills representations from a pre-trained model. In specific, most SSL methods create multiview representations by transforming input images in multiple ways, we instead pursue different representation views from a well-trained data generator.
4.2 Spanning Representations from Synthetic to Real Domain
Here we address the second problem, the domain between synthetic and real domains, due to two factors. First, the synthesized images may of low quality. This aspect has been improved a lot with recent GAN modelling [37, 35] and is out of our concern. Second, more importantly, GAN is notorious for the mode collapse issue, suggesting the synthetic data can only cover partial modes of real data distribution. In other words, the synthetic dataset appears to be subset of the real dataset.
To undermine the harm of mode collapse, we include the real data in the training data of the student network. In particular, in each training step, synthetic data and real data consist of a mini-batch of training data. For synthetic data, the aforementioned squeeze loss is employed. For real data, we employ the original VICReg to compute loss. Specifically, given mini-batch of real data (x" J.y hi= each image x is transformed twice with random data augmentation to obtain two views a, (x(x)) and (x)),, where a;, d' The corresponding representations 7, and Zü, are obtained by feeding the transformed images into Sg similarly to Equ. 8. Then the loss on real data is computed as
Sapa AC + (avar (Z,) Lear (7,)) [oo (7,,) Loov (7,))
(12)
where CRD denotes self-distillation by measuring the distance of two-view representations on real images. The overall loss is computed by simply combine the generated data loss and real data loss as Liotal aLoqussz= + (1 a Capan where 0.5 denotes the proportion of synthetic data in mini-batch of training samples.
From a technical perspective, spanning representation seems to be a combination of representation distillation and SSL using VICReg [3]. We interpret this combination as spanning representation from the synthetic domain to the real domain. The representation is dominantly learned in the synthetic domain and generalized to the real domain. The student network learns to fuse representation spaces of two domains into a consistent one in the spanning process. Our experimental evaluation shows that "squeeze and span" can outperform VICReg on real data, suggesting that the squeezed representations do have a nontrivial contribution to the learned representation.

---

Experiments
5.1 Setup
Dataset and pre-trained GAN Our methods are mainly evaluated on CIFARIO, CIFARIOO, and STL10, ImageNet100, and ImageNet. CIFAR10 and CIFARI00 [39] are two image datasets containing small images at x2x32 resolution with 10 and 100 classes, respectively, and both split into 50,000 images for training and 10,000 for validation. STL-10 [13], which is derived from the T
ImageNet [14], includes images at 96x96 resolution over 10 classes. STL-10 contains 500 labeled images per class (i.e. 5K in total) with additional 100K unlabeled images for training and 800 labeled images for testing. ImageNet100 [55] contains images of 100 classes, among which 126,689 images are regarded the train split and 5,000 images are taken as the validation split. ImageNet [14] is a popular large-scale image dataset of 1000 classes, which is split into I. 1,281, 1,167 images as training set and 50,000 images as validation set. We adopt StyleGAN2-ADA for representation distillation since it has good stability and high performance. GANs are all pre-trained on training split. More details can be referred in the supplementary material.
Implementation details The squeeze module uses linear layers to transform the generator features into vectors with 2048 dimensions, which then summed up and fed into three-layer MLP to get a 2048-d teacher representation. On CIFAR1O and CIFAR 100, we use ResNet18 [30] of the CIFAR variant as the backbone. On STL10, ImageNet, we use ResNet50 as the backbone. On top of the backhone network, a five-layer MI.P is added for producing representation. We use SGD optimizer with cosine learning rate decay [43] scheduler to optimize our models. The actual learning rate is linearly scaled according to the ratio of batch size to 256, i.e. base_lr
ResNet18 the backbone. On ImageNet100 and
batch_sized /266 [24]. We follow the common practice in SSL [7, 55, 29] to evaluate the distilled representation with linear classification task. More details are available in the supplementary material.
5.2 Transferring GAN Representation
Compared methods In this section, we justify the advantage of distilling generator representations by comparing the performance of different ways of transferring GAN representation. In particular, we consider the following competitors:
Discriminator. As the discriminator network receives image as input and is ready for repre- sentation extraction, directly extract features, single penultimate features, multiple features (Eiu 2), using a pre-trained discriminator and train a linear classifier on top of them. Encoding. We train post hoc encoder with without real images involved in the training process as in Equ 3.
Distilliing latent variable. We employ the vanilla distillation or squeeze method on latent variables with data augmentation engaged.
Distilling generator feature. Our method as described in Section 4.
Results Table presents the comparison results, from which we can draw the following conclusions. (I) Representation distillation, whether from the latent variable or generator feature, significantly outperforms discriminator and encoding. We think this is because image reconstruction and realness discrimination are not suitable pretext tasks for representation learning. (2) Distillation from latent variable achieve comparable performance to distillation from generator feature, despite that the former one show entangled class information (Fig. 2). This result can be attributed to projection head the student network. (3) Our method works significantly better than vanilla distillation which does not employ squeeze module. This result suggests that method squeeze more informative representation that can help to improve the student performance.
5.3 Comparison to SSL
Linear classification We further compare our methods to SSL algorithms such as SimSiam [10] and VICReg [3] in different training data domains: real, synthetic, and mixture of real and synthetic.
'https://githun htt.aa/.....wlaa/////gggg.ga/ ada- ada-pytorch
7

---

Knowledge Source
Transfer Method
Domain
CIFARIO
CIFAR 100
Direct use (single feature)
Syn. & Real
63.81
30.11
Discriminator
Direct use (multi-feature)
Syn. & Real
77.58
51.63
Encoding
Syn.
57.15
32.19
Encoding
Syn. & Real Syn.
50.27
28.43
Latent variable
Vanilla distillation (w/ aug)
84.84
53.26
Squeeze
Syn.
86.99
58.56
Squeeze and span
Syn. Real
90.95
66.17
Vamilla distillation (w/ aug)
Syn.
84.48
52.77
Generator feature
Squeeze Squeeze and span
Syn.
87.67
57.35
Syn. Real
92.54
67.87
Table Reprensentation transfer from different teachers. Top-1 accuracy of linear classification on CIFAR10 and CIFAR100 validation sets are reported and compared.
Pretrain Data
Methods
CIFARIO
CIFAR100
STL10
ImageNet100
ImageNet
SimSiam [10]
90.94
62.44
71.30
Real
VICReg [3]
89.20
63.31
74.43
SimSiam [10]
85.11
47.89
73.38
Syn
VICReg [3]
84.68
52.84
70.80
Squeeze (Ours)
87.67
57.35
73.35
SimSiam [10]
90.88
62.68
71.70
Real & Syn
VICReg [3]
90.46
65.22
75.05
46.42
47.32
Sq & Sp (Ours)
92.54
67.87
76.83
53.32
47.80
Table 2: Linear classification performance comparison to seminal SSL methods. Top-1 accuracy on validation set is reported. The bigest number bolded and the second biggest number is underlined.
Table presents the linear classification results, from which we want highlight the following points. (1) Both SimSiam and VICReg perform worse when pre-trained on only synthetic data than only real data, indicating the existence of a domain gap between synthetic data and real data. (2) Our methods outperform SimSiam and VICReg in synthetic and mixture domains, suggesting distillation of generator feature contributes extra improvement SSL. (3) Our "Squeeze and Span" is the best among all the competitors CIFARIO, CIFAR 100, and STL10. (4) Our method outperforms VICReg with a large margin (6.90% Top-1 Acc) on ImageNet100 and a clear increase (0.48% Top-1 Acc) on ImageNet.
Transfer learning As one goal of representation learning transferability to other datasets, we further conduct comprehensive transfer learning evaluation. We follow the protocol in [20] and use its released source code to conduct a thorough transfer learning evaluation for our pre-trained models on 1rtigeeee1000/mmaeeeee
In particular, the learned representations are mainly evaluated for (I) linear classification on datasets including Aircraft, Caltech101, Cars, CIFARIO, CIFARIO0, DTD, Flowers, Food, Pets, SUN397, and VOC2007; (2) finetuning three downstream tasks and datasets, including object detection on PASCAL VOC, surface normal estimation on NYUv2, and semantic segmentation ADEChallenge2016 Please refer [20] for the details of evaluation protocol.
The results are presented Table and Table 4, where we have the following observation (1) As depicted in Table 3, our method achieves better transferability than VICReg the mixed data no matter pre-trained on ImageNet100) or ImageNet. Our method beats VICReg on nearly all other datasets and the improvement average accuracy is 3.40 with models pre-trained and 1.00 with models pre-trained on ImageNet. (2) As depicted in Table 4, representations Iearned with method can be well transferred to various downstream tasks such as object detection,
1nageNet100

---

Pre-training Data
Caltech101
CIFAR100
Flowers
SUN397
VOC2007
Aug.
InageNeti00
VICReg
23,96
60.29
15.27
80.28
57.11
45.95
60.26
33,40
38.41
29.53
49.07
44.86
(Syn,&Real)
Sqj8p (Ouns)
23.88
63.59
15.30
84.37
61.28
49.36
63.41
37.80
43,28
33.04
35.64
48.26
InaggeNot
VICReg
32.39
79:49
22.37
90.09
70.67
58.88
79.97
50.55
59.47
45.76
68.74
59.85
RRyn&&Reah)
Sp8:5p (Ouss)
33,85
80.65
25.71
90.14
70.81
61.75
80.09
50.84
61.01
46.34
68.30
60.86
Table 3: Linear classification performance on 11 downstream classification datasets.
VOC Detection
NYUv2 Surface Normal Estimation
ADE Semantic Segmentation
Pre-training
Method
AP5D
AP75
Mcan
Median
11,25"
f
loU
Accumey f
ImageNet100
VICReg
33.75
61.73
31.77
34.62
29.70
20.90
39.77
50.40
0.3008
73.19
(Syn,&Real)
SqkSp (Ours)
41.10
69.07
42.54
33.09
27.47
22,90
42.71
53.47
0.2993
73.25
ImageNet
VICRog
46
75.78
49.07
33,86
41.35
52.15
0.3263
74,85
(Syn.&Real)
Spt (Ours)
48.98
77.50
52.85
33,39
28,30
22.27
41.65
52.26
0.3299
75.22
Table 4: Downstream task finetuning performance, including object detection on PASCAL VOC, surface normal estimation on NYUv2, and semantic segmentation on ADEChallengee016. t denotes higher is better while + denotes lower is better.
surface normal estimation and semantic segmentation, and consistently show higher performance than VICReg.
We believe these results suggest that generator features have strong transferability and great promise to contribute to self- self-supe representation learning.
5.4 Ablation Study
Effect of squeeze and span The effect of our method is studied by adding modules to the vanilla version of representation distillation (a) one by one. (a) - (b): After added data augmentation, sig- nificant improvement can be observed, suggesting that invariant representation to data augmentation is crucial for linear classification performance. This result inspires us to make teacher representation more invariant. (b) + (c): the learnable TG is introduced to squeeze out invariant representation as teacher. However, trivial performance (10% top-1 accuracy, no better than random guess) is obtained, implying models learn trivial solutions, probably constant output. (c) -> (d) & (e): regularization terms are added, and the student network now achieves meaningful performance, which indicates the trivial solution is prevented. Moreover, using both regularizations achieves the best performance, which outperforms (b) without "squeeze". (e) + (f): training data is supplemented with real data, i.e. adding "span", the performance is further improved.
Domain gap issue We calculate the squared MMD [4] of representations of synthetic and real data to measure their gap in representation space. Table shows "Squeeze and Span" (Sq&Sp) reducesthe MMD compared "Squeeze" by an order of magnitude on CIFARIO and CIFAR 100 and large margin on STL10, clearly justifying the efficacy of "span" as reducing the domain gap.
Impact of generator We further compare the performance of our method when we GAN checkpoints of different quality. Fig. 4 shows the top-1 accuracy with respect to FID, which indicates the quality of GAN. It not surprising that GAN quality significantly im- pacts our method. The higher the quality of generator we utilize, the higher performance of learned representation our method can at- tain. It noteworthy that moderately trained GAN (FID < 11.03) is already able to con- tribute additional performance improvement on CIFAR 100 when compared to VICReg trained on a mixture of synthetic and real data. In the appendix, further analyze the im-
Squeese and span
VICReg on mixture
Topl Acc (9%)
66
VICReg
64
62
4.13 5:22
7.09
11.03
20.78
41.71
87.51
FID (log-scale)
Figure 4: Representation performance (top-1 accu- racy) versus generator quality (FID) on CIFAR100. A better GAN has lower FID.

---

Lun
Ta
Span
Top -l
74,20
Methods
Pretrain Data
CIFARIO
(x100²)
CIFARIO.
(x10
(x10
STL.0
v
84.48
Syn
3.44
5.89
5.39
V
V
10.00
VICReg [3]
Real & Syn
3.74
16.8
11.4
79.
87.67
Squeeze (Ours)
Syn
4.79
1.24
9.82
V
V
92.54
Sq Sp (Ours)
Real & Syn
0.45
0.25
3.71
Table 5: Ablation study on CIFARIO. Ta and A denote whether to introduce the learn- able squeeze module and data augmentation, respectively. CRD. var and Leov represent whether to enable the correpsonding losses.
Table 6: Squared MMD between synthetic and real data representation that measures the discrepancy of representation across domains. Lower number indicates smaller domain gap.
pact of generator feature choices and GAN architectures on the distillation performance.
Conclusions
This paper proposes to "squeeze and span" representation from the GAN generator to extract transfer- able representation for downstream tasks like image classification. The key techniques, "squeeze" and "span", aim mitigate issues that the GAN generator contains the information necessary for image synthesis but unnecessary for downstream tasks and the domain gap between synthetic and real data. Experimental results justify the effectiveness of method and show its great promise in self-supervised representation learning. We hope more attention can be drawn to studying GAN for representation learning.
Limitation and future work The current form of our work still maintains several limitations that need to be studied in the future. (1) Since we distill representation from GANs, the performance of learned representation relies the quality of pretrained GANs and thus limited by the performance of the GAN techniques. Therefore, whether a prematurely trained GAN can also contribute to self-supervised representation learning and how effectively distill them will be an interesting problem. (2) In this paper, the squeeze module sets the widely-used transformation-invariie the learning objective of representation distillation. We leave other learning objectives tailored for specific downstream tasks as future work. (3) More comprehensive empirical study with larger scale left as future work to further exhibit the potential of our method.
as
Acknowledgments and Disclosure of Funding
This work was supported by the National Key R&D Program of China under Grant 201AAA0012200,, National Natural Science Foundation of China under Grant 61620106600 0600000 EPSRC Visual Al grant EP/T028572/1.
References
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In ICCV, 2019,
[2] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. CVPR, 2019.
[3] Adrien Bardes, Jean Ponce, and Yann LeCun. Viereg: aariane-invariane-ccvaaan regularization for self-supervised learning. arXiv preprint arXiv:2105 04906, 2021. 2, 3, 5, 10
[4] Mikolaj Birikowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In ICLR, 2018. 9
[5] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale training for high fidelity natural image synthesis. In ICLR, 2018. 1
[6] Lucy Chai, Jun Jun-Yan Zhu, Eli Shechtman, Phillip Isola, and Richard Zhang. Ensembling with deep generative views. In CVPR, 2021. 4
10

---

[7] Ting Chen, Simon Korbllith Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 3. 7.7
[8] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming Hc. Improved baselines with momentum contrastive learning- arXiv preprint arX1v:2003.04227, 2020. 3
[9] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2020. 3
[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CVPR, 2021. 2, 5, 6.
7,
[11] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. ICCV, 2021.
[12] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, 2019, 3
[13] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, 2.11.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, 2009. 7
[15] Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
[16] Carl Doersch and Andrew Zisserman. Multi-task supe-viee visual learning. In ICCV, 2017. 3
[17] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In ICLR, 2017. 1, 2, 4
[18] Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In NeurIPS, 2019. 1,
2.4
[19] Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. In ICLR, 2017. 2, 4
[20] Linus Eriesson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer?
In CVPR, 2021. 8
[21] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations, In ICLR, 2018.
[22] Lore Goetschalckx. Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual definitions of cognitive image properties. ICCV, 2019. 1,4
[23] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NewrlPS, 2014.
[24] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenct in I hour. arXiv
preprint arXiv: 02677, 2017.
[25]
Jean- Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Flena Buchatskaya, Carl Doersch, Bernardo Avila Pires. Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: new approach to self- supervised learning. In NeaRIPS, 2020. 3
[26] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping.
In CVPR, 2006. 3
[27] Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li. Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 20212 3
2021.
[28] Kaiming He, Hanqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised
visual representation learning. In CVPR, 2020. 3
[29] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised
visual representation learning. In CVPR, 2020. 7
[30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, 2016. 7
[31] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In
NewrIPS Workshop. 2014, 3
[32] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. In NeurtPS, 2020. 1,4
[33] Ali Jahanian, Lucy Chai, and Phillip Isola. On the" steerability" of generative adversarial networks. In ICLR, 2020. 1,4
[34] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lchtinen, and Timo Aila. Training generative adversarial networks with limited data. In NewrlPS, 2020. 1.2
[35] Tero Karras, Mïika Aittala. Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. NewrIPS, 2021. 2, 4, 6
[36] Tero Karras, Samuli I Laine, and Timo Aila. A bayle generator architecture for generative adversarial networks. In CVPR, 2019. 2,
[37] Tero Karras. Samuli Laine, Mila Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020. 1, 4, 6

---

[38]
Jangho Kim, Seonguk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. In NeuRIPS, 2018. 3
[39]
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. 7
[40] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In ECCV, 2016. 3
[41] Yuncheng Li, Jianchao Yang. Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In ICCV, 2017.
[42] Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf. Olivier Bachem, and Michael Tschamnen. superi-superviied disentanglement without compromises. In ICML, 2020.
[43] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2016. 7 Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossherger. Umap:
[44]
Uniform manifold approxi-
mation and projection. JOSS, 2018. 4
[45] Jiteng Mu, Shalini De Mello, Zhiding Yu, Nuno Vasconeclos, Xiaolong Wang, Jan Kautz, Sifei Liu, and
Ue Diego Coordgan: Self-supervised dense correspondences emerge from gans. In CVPR, 2022. 1, 1,2
[46] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw
puzzles. ECCV, 2016.
[47] Deepak Pathak, Philipp Krähenhühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders:
Feature learning by inpainting. In CVPR, 2016. 3
[48]
William Peebles, Jun- Yan Zhu, Richard Zhang, Antonio Torralba, Alexei Efros, and Eli Shechtman.
San-supereised dense visual alignment. In CVPR, 2022. 1,2
[49] Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot. Controlling generative models with continuous factors of variations. In JCLR, 2020. 1,4
[50] Alee Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016. 2,
[51] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2014, 3
[52] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In CVPR, 2020. 1.4
[53]
Nurit Spingarn Eliezer, Ron Banner, and Tomer Michaeli. Gan steerability without optimization. ICLR, 2021. 1,4 Suraj Srinivas and François Fleuret. Knowledge transfer with jacobian matching. In ICML, 2018. 3
[54]
[55] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020. 7
[56] Nontawat Tritrong, Pitchapom Rewatboworwwong, and Supasorn Suwajanakorn. Repurposing gans for
one-shot semantic part segmentation. In CVPR, 2021. 1.2
[57] Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In ICML, 2020. 1,4
[58] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling, arXiv preprint arXiv:211 1.09886, 2021. [59] Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolei Zhou. Generative hierarchical features from synthesizing images. In CVPR, 2021.
[60] Chenglin Yang. Lingxi Xie, Siyuan Qiao, and Alan L. Yuille. Training deep neural networks in generations:
A more tolerant teacher educates better students. In AAAI, 2019. 3
[61] Yu Yang, Xiaotian Cheng, Hakan Bilen, and Xiangyang Ji. Learning to annotate part segmentation with gradient matching. In ICLR, 2022. 1,2
[62] Junho Yim. Donggyu Joo, Ji-Hoon Bae, and Junmo Kim. gift from knowledge distillation: Fast optimization, network minimization and transfer lcarning. In CVPR, 2017. 3
[63] Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: improving the performance of convolutional neural networks via attention transfer. ICLR, 2016.
[64] Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised
learning via redundancy reduction. ICMI. 2021. 3
[65]
Richard Zhang, Phillip Isola, and Alexci A. Efros. Colorful image colorization. In ECCV, 2016.
[66]
Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metrie. In CVPR, 2018. 4
[67] Yuxuan Zhang, Wenzheng Chen, Huan Ling. Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image gans meet differenniable rendering for inverse graphics and interpretable neural rendering. In
ICLR, 2021. 1,2
[68] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean- Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In CVPR, 2021. 1.2 2
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A]. You are strongly encouraged include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Section 5.1 and supplementary for thorough implementation details and the source code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you' rou're
using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Section 5.1
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "CskyedSm2pC", "reviewer": "Reviewer_pYVQ", "summary": "This paper proposes to distill representations from GANs for downstream tasks. Specifically, it emphasizes the importance of generator features and proposes a squeeze and span solution to effectively extract such generator features as image representations. To do so, it learns a feature prediction network that predicts generator features transformed under some kind of semantic-preserving transformations. So that generator features of real images can be directly computed. Moreover, to reduce the gap between real and fake images and avoid the generator representation of a real image suffer from GANs' issues such as mode collapse, the proposed method further applies a spanning operation that trains the feature prediction network also on real images using VICReg. ", "questions": "Overall I think current experiments are significantly insufficient to comprehensively study the effectiveness and different properties of GAN generator features as self-supervised representations. My detailed concerns are covered in weaknesses. I believe studying generator features as self-supervised representations across generator architectures, feature depths, downstream tasks, as well as test datasets will lead to many more insightful observations. ", "limitations": "Authors discussed the limitations in the main manuscript, which look reasonable to me.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The task of distilling GAN features for downstream perception tasks is definitely a worthexploring direction\n\n+ The manuscript is easy to follow\n\n+ The visualization included in Figure 2 well motivates the use of generator features over latent variables and discriminator features.\n\nWeaknesses:\n- one missing related work:\n\n[a] Generative Hierarchical Features from Synthesizing Images, CVPR 2021.\n\nwhich also utilizes generator features for various downstream tasks.\n\n- Experiments should be significantly extended. Specifically, authors only conduct experiments on low-resolution images of CIFAR10, CIFAR100 and SHVN. As pointed out by the authors as the third point of the limitations, the proposed method should be validated on more challenging datasets with higher resolution images. I believe this is a must. Since practical value is an important aspect for self-supervised learning methods, CIFAR10, CIFAR100 and SHVN are insufficient to reflect the richness and informativeness of generator features.\n\n- Authors only used StyleGAN2-ADA as the target generator to distill from, which is not sufficient. Studying the generator features of different GAN architectures is important for studying the effectiveness of generator features since different architectures naturally have different inductive bias.\n\n- Authors only used distilled generator features on the test set of the same dataset the gan is trained on. However, one property of self-supervised methods is that their representations are generally useful across datasets. Authors should include such a study.\n\n- Authors only used distilled generator features on image classification. Other downstream tasks are needed to study the pros&cons of generator features as a self-supervised representation. In fact, maybe authors should follow the standard protocol for evaluating self-supervised methods.  \n\n- Just concatenate all layer features as the representation provided by the generator is insufficient to obtain solid observations. Authors should include studies that use different features or combinations from the single generator. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2su8U9EqP0K", "reviewer": "Reviewer_qg1m", "summary": "The main idea of the paper is how to use the pre-trained generator network of GAN to improve the downstream task of classification. Specifically, the paper use with VICReg-based method to train the network S (called student) that takes the real inputs and applies different augmentations before putting them as inputs of S, the features output of these augmentations is regularized to be similar (invariant to transformation) while the variance-covariance are maximum (this task is called “span” module). Then, the method applies the pre-trained generator to synthesize images and these images are input into the S to produce the features which are regularized from match with those features from concatenation of feature maps (go through module called “squeeze” module before that) that generated the synthetic image. The losses for synthetic images are similar to on real images except the distilled loss between generator’s features and output of the student' feature. The experiments on low resolution datasets (CIFAR-10, CIFAR-100 and STL10) showing the improvements over two existing works SiamSiam and VICReg trained on the real, fake or mixed data. The paper conducted the ablation study to understand the contributions of different components.  \n\nHowever, the paper is written by the other way around: starting with “squeeze” and improve with “span” module that I think want to make the idea more interesting (but probably makes reader a bit confusing on what is actual contribution)?  With this order, the paper investigates different ways to exact features from GAN: from discriminator, latent variable and generator. The 2D visualisation and experiments suggest the features from generator are most discriminative. Inspired from the generator’s features, the paper introduces two modules “squeeze” and “span” to extract the generator features for distillation.  ", "questions": "See above", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n**Strength**\n\nI think the main contribution this paper regarding the \"squeeze\" module and how the generator being used to improve the existing self-supervised models. It is novel enough to me and this brings some improvements (2%) from the simple way of mixing real/fake dataset. The paper is well-written and easy to read. \n\n**Weakness**\n\nFrom my perspective, “span” module is considered as one contribution is questionable as it is just similar to VICReg method. Therefore, I am not sure some claims of the span module to address the mode collapse and the gap between real and fake is convincing enough, except if the authors can bring some theoretical / empirical results to show that. Otherwise, it is not highly motivated why we need the generator for distillation instead of the classifier? \n\nAt the moment, I put as borderline accept but the rating can be increased and decreased depends also comments of other reviewers which may point out something I am missing and also the rebuttal. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-cuVQdTzAQd", "reviewer": "Reviewer_8CJZ", "summary": "The paper proposes a method to distill representations from a pretrained GAN generator for the downstream classification tasks. The paper further proposes a \"squeeze\" technique to make the representation compact, and a \"span\" technique to mitigate the synthetic to real domain gap. Experimental results on small datasets including CIFAR10, CIFAR100, and STL10 show its efficiency in self-supervised representation learning.", "questions": "See \"Weaknesses\".", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__\n\n- This paper is well-written and easy to follow.\n- The idea of distilling representations from GAN Generators is well-motivated.\n- The proposed squeeze and span techniques mitigate the mentioned problems effectively.\n\n__Weaknesses__\n\n- The proposed squeeze and span techniques are heavily inspired by VICReg, which downweights the novelty of the paper.\n- From a technical perspective, distillation from \"latent variable\" and \"generator feature\" are not that very different. This can be verified by the numbers in Table 1, the vanilla distillation and squeeze accuracies are very similar. I also wonder what are accuracies of the \"squeeze and span\" for \"latent variable\" and how would those compare to the proposed full model.\n- It is hard to assess the effectiveness of the proposed method without experiments on ImageNet for such an empirical paper. I understand that the limited computational resources might be a concern, but any form of ImageNet like low resolution of 64x64 or TinyImageNet (which has 200 classes) would be helpful.\n\n__Post Rebuttal__\n\nI thank authors for their response. Some of my concerns have been addressed. But my concern on missing results on large-scale datasets still remain. Therefore, I keep my initial rating but lean towards acceptance for this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept