Optimal Positive Generation via Latent Transformation for Contrastive Learning
Authors: Yinqi Li
Hong Chang
Bingpeng Ma
Shiguang Shan
Xilin CHEN
Venue: NeurIPS 2022 Accept (2022)
Keywords: Contrastive Learning
Self-Supervised Learning
Generative Model
GAN

Abstract: Contrastive learning, which learns to contrast positive with negative pairs of samples, has been popular for self-supervised visual representation learning. Although great effort has been made to design proper positive pairs through data augmentation, few works attempt to generate optimal positives for each instance. Inspired by semantic consistency and computational advantage in latent space of pretrained generative models, this paper proposes to learn instance-specific latent transformations to generate Contrastive Optimal Positives (COP-Gen) for self-supervised contrastive learning. Specifically, we formulate COP-Gen as an instance-specific latent space navigator which minimizes the mutual information between the generated positive pair subject to the semantic consistency constraint. Theoretically, the learned latent transformation creates optimal positives for contrastive learning, which removes as much nuisance information as possible while preserving the semantics. Empirically, using generated positives by COP-Gen consistently outperforms other latent transformation methods and even real-image-based methods in self-supervised contrastive learning.

Optimal Positive Generation via Latent Transformation for Contrastive Learning
Yinqi Li Hong Changl? Bingpeng Ma², Shiguang Shan 1:2 Xilin Chen
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences
yinqi. witphi1..iilliiicc..... ict.pp..ccctteei. ac. {changhong, agshan, xlchen]@ict... cn. bpmaQucas.ac.co
Abstract
Contrastive learning, which learns contrast positive with negative pairs of sam- ples, has been popular for self-supervised visual representation learning. Although great effort has been made design proper positive pairs through data augmenta- tion, few works attempt to generate optimal positives for each instance. Inspired by semantic consistency and computational advantage in latent space of pretrained generative models, this paper proposes to learn instance-specin latent transfor- mations generate Contrastive Optimal Positives (COP-Gos for self-supervised contrastive learning. Specifically, we formulate COP-Gen as an instance speecifcc latent space navigator which minimizes the mutual information between the gen- erated positive pair subject to the semantic consistency constraint. Theoretically, the learned latent transformation creates optimal positives for contrastive learning, which removes much nuisance information possible while preserving the semantics. Empirically, using generated positives by COP-Gen consistently outper- forms other latent transformation methods and even val-image-baase methods in self-supervised contrastive learning.
Introduction
Learning effective visual representation without explicit human annotation is attractive but challenging machine learning and computer vision. One way achieve this goal through self-supervised learning, which leverages handcrafted pretext tasks to create the learning objectives, such as context prediction [1], colorization [2], rotation prediction [3], and generative modeling [4, 5].
Recently, contrastive learning [6. 7, 8, 9, 10, 11] has achieved great success self-supervised visual representation learning. The goal of contrastive learning is pull semantically similar samples (positive pairs) together in the feature space while pushing dissimilar samples (negative pairs) apart. Without access labels, negative pairs are sampled randomly and in large amount [10, 11] obtain effective representations [12, 13]. A line of works are proposed to calibrate the impact of negative samples [14, 15, 16, 17, 18. 19].
On the other hand, the construction of positive pairs is also very crucial. Tian et al. [20] demonstrates that optimal positive pairs task-dependent. Since the labels of downstream tasks and pretraining dataset are not accessible in self-supervised learning, a common way to create positive pairs is to cenaanti-presservngg data space transformation (augmentation) obtain two more different views of the same sample, as shown in Figure la. Great effort has been made to find the optimal augmentation policy [1 20]. Nevertheless, data augmentation cannot create instance-speeiffc optimal positives, which are key in contrastive learning to preserve semantic and discard nuisance information for each sample. Some works try to address this problem by learning adversarial
Code and models available https://iihbb.cmm///iigggee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

agreement
Max agreement
h'
f(-)
f(-)
Max agreement
tx Tx
1
Tx
Tx
{ Tx
f()
x
G(-)
G(-)
G(-)
G(-)
1 ~ Tp
T₂() Min agreement
2
(a) SimCLR [11]
(b) GenRep [28]
(c) COP-Gen
Figure I: Different ways of creating positive pairs for contrastive learning. (a) is based on data space transformation Tx while (b) (c) based on latent space transformation Tz. Our proposed COP-Gen aims to create optimal positives for each anchor while (a) (b) do not. Blue arrows indicate learnable and black non- non-learnable operations. Figure inspired by [11].
perturbations in data space [21, 22, 23] or using nearest neighbors in the learned representation space [24, 25, 26, 27J. However, the former limits the transformation diversity and the latter suffers from high false positive rate [25].
Instead of creating optimal positives for each instance in data space or representation space, more proper way to address this problem through transformation in the latent space of well-trained generative models. The main reasons include (1) semantic consistency between the positive pair can be achieved in the latent space (in contrast representation space transformation and non-domain- knowledge guided data augmentation) by controlling the transformation distance within a small range; (2) latent space transformation continuous and differentiable (in contrast data augmentation), which is computationally friendly for searching instance-wiss optimal transformations. In addition, may bring more diversity beyond data augmentation. Jahanian al. [28] have made first step towards leveraging latent space transformation together with data augmentation to create positive pairs, as shown in Figure lb. However, the transformation is still designed for the whole dataset. How to find the optimal latent transformation for each sample remains open issue.
In this work, address the above problems, we propose an instance-specific Contrastive Optimal Positive Generation (COP-Gen) approach via latent space transformation. Specifically, COP-Gen learns a navigator in the latent space, whose objective to transform each input anchor to the optimal position. To this end, subjecting to the semantic constraint, the navigator is simply learnt to minimize the mutual information between the generated positive pair. Therefore, the training process is implemented by adversarially training mutual information estimator together with the latent space navigator, as illustrated Figure lc. We summarize our contributions as follows:
We find the latent space of well-trained generative model fits exactly for generating instance-specific optimal positives in fully unsupervised setting and thus propose COP-Gen.
We theoretically demonstrate that the learned latent transformation generates optimal positive pairs for contrastive learning, which can preserve semantic-relevant information while discarding nuisance information.
Extensive experiments show that using COP-Gen to generate positives outperforms other latent transformation methods and even real-image-bare based methods in self-supervised contrastive learning. We also extend COP-Gen supervised contrastive learning.
2 Preliminaries
Contrastive Learning. Given training sample state-of-the-art contrastive methods like SimCLR [11], as illustrated Figure 1a, use data augmentations lx Tx and i' ~ Tx to create positive pair (v, v'), and then learn encoder f to maximize the agreement between them by contrasting

---

with K randomly sampled negatives, using the InfoNCE loss [9]:
(p)(sim(f(v:
(V v') = E log exp(sim(f(v).f(v))) + Ex= exp(sim(f f(v()))
(1)
where sim denotes the cosine similarity between two vectors. is proved that minimizing InfoNCE loss maximizes lower bound on mutual information [9, 29]: v(v; 2 log(K) -L where v(v:v') represents mutual information between v and v'.
Optimal Positives for Contrastive Learning. Intuitively, the data augmentation Tx that creates positive pairs should preserve the semantic characteristics while changing the nuisance aspects [30]. To find the right balance, Tian et al. [20] introduce "InfoMin principle", which demonstrates that optimal positives for contrastive learning are task-dependent.
Definition 2.1 (Optimal Positives for Contrastive Learning, Proposition A.I in [20]). Given downstream task T with label y, the optimal positive pair (v",'") created from the data x satisfies
I(v";v'*) = y(x;),, I(v":v'"y) = 0,
subject to [(v:y) = I(v'yy) [(x;))
Definition 2.1 indicates that when we have access to the downstream task, the optimal positive pair should only capture the task-relevant information / y)) while leaving the nuisance information I(v";v'(y) ly) to be zero.
Unfortunately, y is not accessible in unsupervised contrastive learning. The construction of positive pairs commonly guided by domain knowledge or experimental results instead [11, 20] and could not be instance- sastance-wise optimal. To address this problem, we will next develop an optimal positive generation approach that uses pretrained generative models without accessing the labels.'
Contrastive Optimal Positive Generation via Latent Transformation
In this section, we first introduce contrastive optimal positive generation (COP-Goy approach in Section 3.1. Next, we give a theoretical analysis in Section 3.2. In Section 3.3 we describe and explain the practical details.
3.1 Motivation and COP-Gen Approach
Before describing our approach, we explain the inspiration behind it by formulating a prominent property of pretrained generative models. It has been observed that using pretrained generative models can create images with interpretable transformations [31, 32, 33], which has several interesting applications such as face attribute editing [34] and data augmentation [35]. These generative models [31, 36, 37] are commonly trained on annotated or one-class datasets. Yet is worth noting that even they are trained unlabeled large scale dataset like ImageNet [38], nearby latent vectors can map to semantically similar images, which is remarkable property for unsupervised representation learning [28]. See Figure 2 for an empirical observation. We formulate this property as follows.
Proposition 3.1 (Semantic Consistency in Latent Space). Given a well-trained unconditional gener- ative model G, z and z' are latent vectors, if the distance d(z, z') S 8, then G(z) and G(z') will have the similar semantic label y. In the form of mutual information, we have
[T((()):)) - T(G((z);))) Se,
(4)
where stands for tolerable semantic difference, and 8 is the maximum shifted distance to maintain semantic consistency (approximately corresponding to the second column of G(z') in Figure 2).
The above observation and Proposition 3.1 inspire to find optimal positives in the latent space of well-trained generative models without access to the labels. Therefore, we propose the following
'The optimal positives for contrastive pretraining always depend on downstream task, which is usually unknown in advance. We here focus on addressing the inaccesssee issue of pretraining dataset, which is assumed to be helpful and general.
3

---

(() d(a,)))
C(3
((+). d(x,))
Figure 2: Randomly sampled images from ImageNet-pretrainge unconditional BigBiGAN [39]. From left to right for each example, the distance between z and z keeps rising.
COP-Gen approach. Our goal is to learn latent space navigator (2),,, which can map each sampled anchor z to the right position z', where the constructed pair (G(z), G(z')) is the optimal for contrastive learning.
COP-Gen. Our idea, inspired by the InfoMin principle, is to reduce as much shared information between G(z) and GzTz(z)) possible while preserving the semantics:
min / ((x((((x))) & (Z(T)))))), s.t. d(z, (z)) s 6, where z ~~ Pr, Lxs 1/x ~ Tx. Tx
(5)
In practice, a shared encoder ʃ trained with InfoNCE loss applied to estimate a lower bound the mutual information between the positive samples, illustrated in Figure 1c. Specifically, Eq. (5) is implemented as the following adversarial training form:
max min (" T.
s.t. d(z, Tz(z)) S 8, where z Pa; txx Tx (6)
After learning Tr. we freeze it and use it together with G generate positive pairs offline. After that, we perform the standard contrastive learning by training an encoder that maximizes (a lower bound on) the mutual information between generated positive pairs. As will be proved in Section 3.2, there is no nuisance information remaining in any generated positive pairs to be learned by the encoder.
Note that the synthetic images usually object-centric and color-biased [40]. As consequence, when using these synthetic images for contrastive learning, data space augmentations Tx like random cropping and color jittering cannot be replaced even when the latent transformation is optimal. Thus, Tx is also adopted in COP-Gen to keep consistency with the contrastive learning stage.
3.2 Theoretical Analysis
In this subsection, we theoretically demonstrate that under the guarantee of semantic consistency, COP-Gen will find optimal latent transformations for each sampled latent vector. We first give the following assumption and lemma.
Assumption 3.2 (Semantic- (Semantic-preserring Tx). The data space augmentation Tx : X + should be designed to be cemantic-preserviin In the form of mutual information,
[(tx (x):)) (xx y), Vtx ~ Tx:
where x E X' the data, y E y is the semantic label of x.
In practice, we take the well-trained BigBiGAN [39] generator the data source, which was trained jointly with an additional encoder to learn the inverse mapping as BiGAN [41] and ALI [42]. has been proved that these models have the following property.
Lemma 3.3 (Theorem in [41]). Assume discriminator D, generator and encoder are optimal, if G and E are deterministie, then G-1 = E almost everywhere.
Lemma 3.3 ensures the invertibility of the pretrained generator G. In COP-Gen (Figure Ic), the invertibility ensures CZT2 can be mapped back to deterministic place Tg (2(z) in the latent space. Thus training a navigator T2 z + Tz(z) can make sense. Otherwise the navigator would be confused if there were more than one destination. The formal proof will be given in Theorem 3.4. We also use a pretrained flow model [43], which is naturally invertible, as the data source and conduct a toy experiment in Appendix D.
It may not hold for other typical GAN generators.

---

Theorem 3,4 (Optimal T2). Under the above assumption and lemma, given specific G(z). T
obtained by optimizing Eq. (5) generates optimal positives G(T;, (z)) for contrastive learning.
Proof. Below we use the notations in Figure Ic, i.e., z' = _T) x G(z). = G(z'),
bx(x), v = (xx(xx) (x'), for brevity. Using the chain rule mutual information, the training objective in
Eq. (5) can be written as
I(v:V') = [(v:y) - [(v:y/v) + I(v:v'ly)
(8)
We first prove that v introduces no extra semantic information when v' is known, i.e. the above second item
vly') 0.
Strictly, the equality approximately holds within the error range of e as shown below.
Using the chain rule of mutual information, we have
x(y: = [y; + [(y - v'y
(10)
Since 1 preserves the semantics of x', and the semantic consistency between and z' (2(z) holds under the condition of d(z, Ty (z)) S 5 from Proposition 3.1, we have
xy::)) [(y:v)) = [(y:x) I(y: & (x )) = [) - x(y:'')l SE
(11)
by combining Eq. (7) and (4). In addition, under Lemma 3.3, x' and thus v' 1'x (x') can be expressed as a function of x, so that
1(y; 0. By plugging Eq. (1I) and (12) into Eq. (10), find xy: is approximately zero within the error range of s. And v = (x(x) ) is a function of x. Therefore, + v x + v forms a Markov chain approximately. Therefore, Eq. (9) holds within the error range of
(12)
Now, Eq. (8) becomes
T(v:v)= = I(::)) ([v::))+[v;;+)))))
(13)
Since ix is comantic-preserving the first item I(v:y) in Eq. (13) equals to the constant I(x; y). Therefore, minimizing Eq. (13) will make the second item, which represents the nuisance information, tend to zero. Formally, under the semantic constraint, T% obtained by minimizing Eq. (13) satisfies
1(v:v'*) + I(x:y).
(14)
I(v:v'+ly (y) + 0, s. L. d(z, (z)) 8,
(15)
within the semantic error defined in Proposition 1, where v* (GG(TT (z)). According to Definition 2.1, TV creates optimal positives for contrastive learning in latent space.
3.3 Practical Details
Choice of Tz. The latent space navigator T2 is implemented as a nonlinear two-layer neural network T2(z) W2(o(W12 + bı)) + by by default, where a is the ReLU [44] function. We adopt nonlinear trajectories in latent space since linear trajectories may cause generated images distorted or meaningless [45, 46]. Linear Tg is also experimented as an ablation Section 4.3.
Implementation of diz,z') < 8. In practice, we model z' as z' = z + (2(2) instead of z' = Ts (z), since random initializing the parameters of T2 may guarantee d(2, 2'),
thus violate the semantic constraint. Take 1.2 distance as an example, d(z, z') now becomes
dz(2,z') VEK (Iz(Z)))2. Since E R is usually sampled from an isotropic Gaussian 1), (z) approximately (due to nonlinear T2) follows Gaussian distribution with marginals
(z)z); ~ Nuu,, o.), where b4 and σ determined by W1, W2, b,, b2. Thus Eidg (z,z'zz)
(ut + af). Therefore, in order to satisfy d(z, x') S 8 experimentally. we control the initialization of the parameters in Tx and monitor the mean and standard deviation of (2(2) as well as the quality of generated images during training. A summary of the implementation
Σ ((({,()))))
of the semantic consistency constraint during the whole training process is discussed in Appendix B.
5

---

4 Experiments
In this section we conduct experiments on the proposed COP-Gen approach and compare it with other latent transformation and real-image-based methods. We use Pytorch [47] for all experiments.
4.1 Experimental Settings
Pretrained Generative Models. Most of our self-supervised experiments are conducted using Ima- geNet ILSVRR-2012 [38] pretrained BigBiGAN [39, 48]. We do not use the encoder of BigBiGAN, but only use the generator as a black-box data source. Besides, some additional self- eeffsuuerviied exper- iments leveraging MNIST [49] pretrained Residual Flow [43] model can be found in Appendix D. We also extend our approach to supervised contrastive learning [50] in Appendix E.
COP-Gen Training. The latent space navigator T2 implemented as a nonlinear two-layer MLP by default. The sizes of the input, hidden and output layers are the same, all equal to the latent dimension (120-d) of BigBiGAN. Data space SimCLR [11] augmentation is adopted, which is implemented to be differentiable by applying torchvision, transf orms to tensors x on-the-fly, to let gradients flow from v (x) to x. The backbone of the mutual information estimator f is ResNet-18 [51] to save GPU memory. We use InfoNCE loss [9] as the adversarial training objective for f and Tu, with the temperature set to 0.1. Adam [52] (31 = 0.5. 82 0.999) is used as the optimizer, where the learning rate is set to 3 x 10-5 for f and 1 x 10 for T2. We train over 200 generated samples* with batch size of 176, which takes about hour on 4 NVIDIA 2080 Ti GPUs.
Methods for Comparison. We compare our COP-Gen with the following positive generation methods. The baseline SimCLR [1]], which leverages data space transformation (e.g., random cropping. color jittering, etc.) to create positive pairs. InfoMin Aug. [20]. which is inspired by the InfoMin principle and adopts additional (cf. SimCLR Aug.) data augmentations. NNCLR [25], which uses nearest neighbors in the learned representation space as the positives. Negative Feature Transformation [53], which creates hard negatives through feature interpolation. Two latent space transformation methods introduced in GenRep [28]. The one is through Gaussian transformation, with standard deviation set to 0.2 determined by grid searching. The other a "steering" method introduced in [54], which learns a shared latent trajectory over 160K samples to mimic data space transformations (including zoom, shifts, rotations, and color transformations), and then randomly samples the length of the learned trajectory to create positives. More implementation details are given in Appendix C.
Comparison of Computational Overhead. When comparing with non-learning-base latent trans- formation methods (GenRep Gaussian [28]), our COP-Gen introduces additional navigator training overhead. This cost also happens in the previous caaming-bassd latent "steering" method [28]. When comparing with real-image-basee positive sampling in the latent space of generative models are considered. All these computations are not too much (several GPU hours). But if we take training generative models into consideration, the extra training cost would be expensive (typically several GPU weeks). Nevertheless, the community has released many well-trained generative models for researchers to use. The detailed computational overhead is listed in Appendix F.
baseline [11], 1] all computations including navigator training and
Contrastive Learning. We conduct contrastive learning on both real and synthetic datasets. For both of them in all experiments, we use 128 x 128 image size, which is the output resolution of the pretrained BigBiGAN generator. The real image encoder trained on unlabeled ImageNet [38] which has ~ ~..28MM images of 1000 classes (ImageNet- IK), and the synthetic image encoder is trained on 1.3. randomly sampled anchor images from BigBiGAN. We use InfoNCE as the loss, which is optimized using SGD with momentum of 0.9, the learning rate of 0.03 x BatchSize/221 2566, and weight decay of 10-4 We train with batch size of 224 for 100 epochs and decay the learning rate using the cosine decay schedule [55], which takes ~ days for ResNet-50 on 2 NVIDIA 2080 Ti GPUs.
'See more discussions on the number of training samples (i.e. when to terminate the COP-Gen training process) in Section 4.3 and Appendix B.

---

Table 1: ImageNet- IK linear evaluation and PASCAL VOC object detection with different self-
supervised contrastive positive generation methods. All the reported are our reproduced results.
ImageNet Linear Evaluation
VOC Object Detection
Method
Tx
Tx
Top-1
Top-5
AP
APs0
AP7s
Training on ImageNer- IK real images:
SimCLR [11]
N/A
SimCLR Aug.
49.44
75.58
52.91
78.68
58.51
Neg FT [53]
N/A
SimCLR Aug.
50.82
75.09
51.08
77.73
55.05
InfoMin [20]
N/A
InfoMin Aug.
50.93
75.87
51.11
77.58
55.55
NNCLR [25]
N/A
SimCLR Aug.
51.97
76,80
51.70
77.93
56.27
Training on BigBIGAN synthetic images:
GenRep [28]
Nome
SimCLR Aug.
41.63
66.57
51.05
77.22
55.77
NNCLR [25]
None
SimCLR Aug.
42,46
66.46
50.40
76.74
55.00
InfoMin [20]
None
InfoMin Aug.
42.83
67.52
50.18
75.68
54.71
Neg FT [53]
None
SimCLR Aug.
43.26
67.21
50.45
77.11
54.88
GenRep [28]
Gaussian
SimCLR Aug.
48.73
73.13
50.20
77.03
54.37
GenRep [28]
Steering
SimCLR Aug.
51.19
74.97
51.42
77.96
56.43
COP-Gen (ours)
Optimal
SimCLR Aug.
53.25
77.16
53.09
78.95
57.99
4.2
Experimental Results
Linear Evaluation ImageNet-IK. We train fully connected linear classifier on top of the frozen 2048-d embeddings from the pretrained ResNet-50 encoder. is trained using SGD with momentum of 0.9, batch size of 224 for 60 epochs with cosine learning rate decay. Following [28], the initial learning rate is set as 0.3 BatchSize/2555 for the real and 2 x BatchSize/ 256 for the synthetic respectively. We report Top-1 and 5 classification accuracies ImageNet- IK validation set.
The comparison result presented Table 1.* First, COP-Gen achieves the best performance compared to other synthetic-image based methods, demonstrating the effectiveness of our latent space optimal positive generation approach. In addition, it worth noting that synthetic-image-bane basee contrastive representation learning outperform the real-image-basee method, as long the proper latent transformation method is adopted. That is because the latent transformation brings more diversity to and could achieve instance-wise optimal positives. Although there is still gap in the resolution and quality between synthetic and real images, this phenomenon inspires us to further develop black-box generative models datasets to perform visual representation learning.
Object Detection on PASCAL VOC. We test the pretrained encoders on the downstream object de- tection task to evaluate the feature transferabilityy Following the protocol in [10], we use detectron2 [56] to train a Faster-RCNN [57] with the R50-C4 backbone. We fine-ture tune all layers end-to-end with batch size of 4 for 96K iterations (~233 epochs) on the PASCAL VOC [58] trainval07+11 set and evaluate on the test07 set. The results are Table It can be seen that COP-Gen achieves consistent gains over other synthetic encoders and competitive performance compared with real-image-basse baselines.
Transfer Learning and Semi-Supervised Learning Evaluations. Following the procedure in [11, 25]. we evaluate the pretrained encoders on transfer learning and semi-supervised learning tasks. For transfer learning task, we train a linear classifier on following datasets: Food 101 [59], CIFARIO [60], CIFARI00 [60], SUN397 [61], Pets [62], Caltech- 101 [63], and Flowers [64]. For semi-supervised learning task, we follow the setting in [11], i.e., sample I % or 10% of the labeled ImageNet training set class-balanced way, and fine-tune the whole network on the labeled data. These two experiments are implemented based on the open-source codebase VISSL [65] using its standard set hyper-parameters and evaluation protocols. We compare our method with the baseline SimCLR and two of the most competitive methods: NNCLR the real and latent "steering" method. encoders outperform
Table shows the results. is worth noting that synthetie-image-traine real-image- amage-trained encoders nearly all transfer and semi-supervised benchmarks. And our method achieves competitive performances when compared with the previous latent "steering" method.
*The gap between our reproduced results and those on real images is because we use smaller image size, batch size and different optimizers. All these details are clarified in Section 4.1. Results with standard deviations after running experiments multiple times and longer contrastive learning given in Appendix C.3.
7

---

Table 2: Comparison of transfer learning and semi- superiise learning performance. For transfer classification we report Top-1 accuracy. For semi cemi-supereised learning we report Top-5 5 accuracy.
Semi- Semi-Supervised Label Fraction
Linear Transfer Classification
Method
Food101 CIFARIO CIFARI00 SUN397
Pets Caltech- 101 Flowers
1%
5%
Training ImageNet-IK I real images:
SimCLR 1111
65.16
84.44
60.99
66.26
69.72
81.96 82.45
82.81
45.77
77.34
NNCLR [25]
62.53
84.10
60.43
64.49
70.35
82.84
44.09
76.28
Training on BigBiGAN synthetic images:
GenRep Stecring [28]
66.17 17
84.23
63.29
62.46
69.94
84.65
84.78
52.33
78.70
COP- Gen (ours)
67.05
84.09
62.44
63.82
73.56
84.63
84.36
51.84
78.81
4.3
Ablation Study
In this section present thorough analysis of COP-Gen and its influence contrastive learning.
Default Setting. We conduct ablation experiments on ImageNet-100, subset of ImageNet-IK. as done in [66, 15, 28]. In this setting, the syn- thetic encoder is trained on 130K anchor images sampled from pretrained BigBiGAN. Note that these sampled images actually come from 1000 classes since the fixed BigBiGAN was trained on unlabeled ImageNet- Therefore, we here only perform contrastive learning on these syn- thetic images and report ImageNet- 100 linear evaluation results, since training real encoder will be an unfair compari- son. Here, the number of contrastive pretraining epoch set to 200, and other training hyper- the same those
Table 3: ImageNet-100 100 linear evaluation on pre- trained encoders based on synthetic images of dif- ferent ways of positive generation.
Method
Tx
Tx
Top-1
Default setting:
GenRep [28]
None
SimCLR Aug.
58.26
GenRep [28]
Gaussian
SimCLR Aug.
62.62
GenRep [28]
Steering
SimCLR Aug.
63.08
on ImageNet-
COP-Gen (ours)
Optimal
SimCI. .R Aug.
63.96
Contrustive pretraining wo data augmentation:
parameters and process in Section 4.1 by default. The ImageNet- linear evaluation results of the default setting are presented the upper part of Table 3.
GenRep [28]
Gaussian
None
28.90
GenRep [28]
Steering
None
28.68
COP Den- (ours)
Optimal
None
37.40
Do We Still Need Data Augmentation Tx? Having obtained the optimal latent space transformation Tz, a natural question is that do we still need data augmentation Tx when performing contrastive learning. We answer this question by removing Tx and using only T2 to create positive pairs. The results are shown the bottom part of Table 3. It can be seen that in the absence of Tx, the performance drops significantly, even though Tg obtained by our COP-Gen is optimal. And best result achieved by combining eestance-speciffi latent (COP-Gen) and data space transformations.
Further, we conduct ablations by removing color, crop, and all transformations in the data space respectively at both navigator learning and contrastive learning stages, to see whether the learned Tz (together with the frozen G) could cover these data space transformations. As shown in Table 4, the large performance drops imply that G+T,(-)) has poor ability to cover color and crop augmentations, which are crucial when creating contrastive pairs [11]. The reason behind this is probably because im- ages generated by GANs are usually object- centric and color-biased [40]. To overcome this, might involve these complex T& when train- ing generative models G. or train G together with navigator Tz. that ((T)()) could cover more Tx-
Table 4: Ablation Tx Navigator training and contrastive learning adopt the same policy.
T- Top-1
SimCLR Aug.
w/o color
w/o crop
None
63.96
55.26
43.84
36.16
Number of Training Samples for Tg. By default, we decide when to terminate COP-Gen training pro- cess by monitoring the quality of generated positives. This allows efficient experiments in Section 4.2 without operations like grid searching. Now, we increase the number of training samples to investigate its influence on the generated positives and contrastive learning performance. For random anchor im- ages, show how the generated positives evolve over the training time in the middle part of Figure 3.

---

It can be seen that: (1) In the first row, the clothes of the person change and the back- ground becomes simpler and simpler. On the contrary, for an anchor with simple back- ground (row 2), the training process makes it gradually more complex. This phenomenon shows that the learned positives are instance- specific. In addition, the COP-Gen training process only changes the above nuisance in- formation between the positive pair, while keeping the semantics (the person and the obelisk) unchanged, so that the following contrastive learning stage will not focus the semmatic--rrelevant information. (2) As for the last positive in row 4, the semantics of the subject is changed (a human face ap- pearing on the cup), which indicates that the semantic consistency constraint is violated and the training process should have been terminated.
Opeimal
Trining
Sumples 60EK
Anchee
200R
400K
Ganzeion
Stoering
Figure 3: Middle: Generated anchor-specific positives by our COP-Gen approach with respect to the number of training samples. A smoother evolution with more examples is given in Figure in Appendix. Right: Results by other latent-transfonnn transformation-basse positive generation methods.
The contrastive learning results using these positives generated at different training periods are shown in Ta- ble 5. Note that although the best performance may not be achieved through simply and conservatively monitor- ing the quality of generated positives, the results only fluctuate slightly and all of them outperform other non- instance-specific latent transformation methods.
Table 5: Ablation on different numbers of COP-Gen training samples.
Training Samples
200K
400K
600K
Top-1
63.96
64.12
63.56
Linear Tz. We also conduct experiments on modeling T₂ as a linear trajectory. With such design, we achieve 61.18% Top-1 accuracy, which falls behind non-lineat T2 and other latent transformation methods. This may be due to the limited modeling capability of linear transformations.
4.4 Visualization Analysis
We give a visual comparison on the generated positives with other latent transformation methods. As shown in Figure 3, the "steering" method [54, 28]. which learns shared latent trajectory with random scales mimic data augmentations, causes unrealistic in few images. This is partly because the pretrained GANs could not achieve these strong data space transformations. Relatively, the Gaussian transformation constructs positives conservative way after grid searching standard deviation [28], which only ensures the semantic consistency but lacks diversity. In contrast, our COP-Gen is aware of the -nstafce-ppecffie nuisance information and removes that between the positives.
Related Work
Contrastive Learning. The goal of contrastive learning is to contrast positive with negative pairs [6]. Without access to labels, positive pairs are typically created from a single image through data augmentation while negative pairs are sampled randomly from the whole dataset [10, 11]. Many works have been proposed to improve this strategy by correcting the sampling bias mining hard samples. For negative pairs, some works [14, 19] eliminate the false negatives in randomly sampled pairs. Some works construct hard negatives from cxisting samples [15, 24, 17, 16, 53], adversarial perturbations [21, 18]. For positive pairs, Chen et al. [11] and Tian et al. [20] revisit common data augmentations and Reed et al. [67] searches augmentation policies. To construct hard positives, feature extrapolation [53] and neighbor discovery [24, 25, 26, 27, 68] are adopted.
Our goal is more than mining hard but to create optimal positives. The most relevant works to ours are [69, 20] that share the same high level idea to construct positives with minimum sufficient information. However, there are several fundamental differences. The work [69] focuses graph contrastive learning and the model design relies on domain knowledge, while our target is general visual representation learning without any supervision. Besides proposing the InfoMin principle,

---

Tian et. al [20] also conducted view learning experiment to find optimal positives in data space. However. the construction is under the color-channel splitting framework and requires the labels of downstream task. In contrast, COP-Gen finds instance-specific optimal positives in the latent space of pretrained unconditional generative models, which is fully unsupervised setting.
Besides, the objective in [70, 71] simply pulls positives together without explicitly contrasting with negative samples. We check it is possible to combine our contrastive optimal positive generation method with these non-contrastive frameworks in Appendix C.4.
Using Generative Models Data Source. Deep generative models have been developed to produce photo realistic images [36, 37, 72, 731, thus they can be used as great tools for generating desired image datasets. Some works investigate intermediate GAN representations to construct part segmentation datasets [74, 75], while some treat pretrained GANs as black-box models and use them to augment data for robustness [76, 35] or ensembling [77]. The pretrained generative models can be data source alone. Some works [78, 79] leverage class-conditional generative models to train a classifier, while GenRep [28] and this paper focus on a more general setting of visual representation
learning.
Latent Space Navigation. A key technique for using pretrained generative models to produce interested images is latent space navigation [32, 33, 80, 81, 46, 82, 83), which manipulates images by discovering interpretable directions in the latent space. Different from these works, our goal is to learn not only the direction but also the magnitude of the trajectory. In addition, the trajectory is restance-speciffe i.e., the learned transformations applied to different images have different meanings. Refer to Figure 3 for the visualization result.
6 Discussion
In this paper, we propose an instance-specific contrastive positive generation approach based on well- trained generative models. Thanks to the attractive property of the latent space, we simply optimize a latent trajectory that reduces as much information as possible within a reasonable magnitude. Theoretical analysis and experimental results demonstrate the effectiveness of our approach.
Limitations and Broader Impact. Since this work uses synthetic data to perform contrastive learning, the performance naturally and partly depends on the quality of pretrained generative models. Nevertheless, having seen that learning from synthetic dataset has outperformed learning from the real, which might impact representation learning community develop more proper generative models (e.g., to reduce the reliance on data space augmentations) as dataset sources to perform contrastive learning.
whole, are inspired to further
Besides, learning from synthetic data alleviates several societal issues of learning from real datasets, which might be private [84] or biased [85]. On the other hand, pretrained generative models may still leak [86] or inherit the biases from the training data [87]. These risks and impacts on representation learning need to be considered when leveraging pretrained generative models as the data source.
Take a class- imbalanced setting for example. Although vanilla contrastive learning can learn more balanced feature space than its supervised counterpart since it does not use labels [88], it still could not immune to imbalance. is hypothesized that the high-frequency "classes" might dominate the learning process and leaves the low-frequency "classes" under-learned [89]. We here hypothesize that the learning of samples from oowffreueecy "classes" might benefit from COP-Gen generated positives. The training process of COP-Gen has seen the overall imbalanced distribution thus has the potential to take it into consideration to create more proper positives.
Acknowledgements
This work is partially supported by National Key R&D Program of China (No. 2017YFA0700800) and National Natural Science Foundation of China (NSFC): 61976203, 61876171. We thank Dr. Xiaoyi Yin for valuable suggestions on the theoretical part of this work. We also thank anonymous reviewers for their constructive feedback.
10

---

References
Carl Dnersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning hy context prediction. Proceedings of the IEEE International Conference on Computer Vision, pages 1422-1430. 2015.
[2] Richard Zhang, Phillip Isola, and Alexci A Efros. Colorful image colorization. In European Conference Computer Vision, pages 649-666, 2016.
[3] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. International Conference on Learning Representations, 2018.
[4] Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. International Conference on Learning Representations, 2014.
[5] lan Goodfellow, Jean Troget-Abaddi Mehdi Mirza, Bing Xu, David Warde- Fardey, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
[6] Raia Hadsell, Sumit Chopra, and Yann LeCun, Dimensionality reduction by learning an invariant mapping. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2, pages 1735-17433 2006.
[7] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox Discriminative unsupervised feature learning with convolutional neural networks, In Advances in Neural Information Processing Systems, pages 766-774, 2014.
[8] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742, 2018.
[9] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03745. 2018.
[10] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9729-9738, 2020.
[11] Ting Chen, Simon Korblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607, 2020.
[12] Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual infor- mation maximization for representation learning. In International Conference on Learning Representations. 2020.
[13] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. International Conference on Machine Learning, pages 9929-9939, 2020.
[14] Ching-Yan Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefamie Jegelka. Debiased contrastive learning. In Advances in Neural Information Processing Systems, pages 8765-8775, 2020.
[15] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In Advances in Neural Information Processing Systems, pages 21798 21809, 2020.
[16] Joshua David Robinson, Ching-Yao Chuang. Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2021.
[17] Mike Wu, Milan Mosse, Chengxu Zhuang, Daniel Yamins, and Noah Goodman. Conditional negative sampling for contrastive learning of visual representations. International Conference on Learning Representations, 2021.
[18] Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. AdCo: Adversarial contrast for efficient learning of unsupervised representations from telf-trained negative adversaries. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1074-1083, 2021.
11

---

[19] Tri Huynh, Simon Kornblith, Matthew R Walter, Michael Maire, and Maryam Khademi. Boosting contrastive self-supervised learning with false negative cancellation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pages 2785-2795, 2022.
[20] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive ? In Advances in Neural Information Processing Systems, pages 6827-683 0877-663,.220 2020.
[21] Chih-Hui Hui Ho and Nuno Nvasconcelos. Contrastive learning with adversarial examples. In Advances in Neural Information Processing Systems, pages 17081-1709) 2020.
[22] Jiang. Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. In Advances in Neural Information Processing Systems, pages 16199-16210, 2020.
[23] Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning. In Advances in Neural Information Processing Systems, pages 2983-2994, 2020.
[24] Feng Wang, Huaping Liu, Di Guo, and Sun Fuchun. Unsupervised representation learning by invariance propagation. In Advances in Neural Information Processing Systems, pages 5510-3520, 2020.
[25] Debidatta Dwihedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanct, and Andrew Zisserman. With little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588-9597, 2021.
[26] Mehdi Azabou, Mohammad Gheshlaghi Azar, Ran Liu, Chi-Heng Lin, Erik C Johnson, Kiran Bhaskaran- Nair, Max Dabagia, Bernardo Avila-Pires, Lindsey Kitchell, Keith B Hengen, et al. Mine your own view: Self-supervised learning through across-sample prediction. arXiv preprint urXiv:2102.10100 2021.
[27] Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Mean shift for sple-super learning. Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 10326- 10335, 2021.
[28] Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. In International Conference on Learning Representations, 2022.
[29] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171-5180, 2019.
[30] Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf. Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In Advances in Neural Information Processing Systems, pages 16451-16467, 2021.
[31] Alec Radford, Luke Metz. and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Representations, 2016.
[32] Lore Goetschalckx, Alex Andonian, Aude Oliva, and Phillip Isola. GANalyze: Toward visual definitions of cognitive image properties. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 5744-5753, 2019.
[33] Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the GAN latent space. In International Conference Machine Learning, pages 9786-9796, 2020.
[34] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of GANs for semantic face editing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 9243-9252, 2020.
[35] Chengzhi Mao, Augustine Cha, Amogh Gupta, Hao Wang. Junfeng Yang, and Carl Vondrick. Generative interventions for causal learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3947-39557 2021.
[36] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
[37] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401 4410, 2019.
12

---

[38] Olga Russakovsky. Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3)2211225) 2015.
[39] Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Advances in Neural Information Processing Systems, pages 10542-10552, 2019.
[40] Minyoung Huh, Richard Zhang, Jun Zhu, Sylvain Paris, and Aaron Hertzmann. Transforming and projecting images into class-conditional generative networks. In European Conference on Computer Vision, pages 17-34, 2020.
[41] Jeff' Donahue, Philipp Krähenbüüh,, Trevor Darrell. Adversarial feature learning. In International Conference on Learning Representations, 2017.
[42] Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martin Arjovsky, Olivier Mastropietro. and Aaron C. Courville. Adversarially learned inference. In International Conference on Learning Representations, 2017.
[43] Tian Qi Chen, Jens Behrmann, David Duvenaud, and Jöm-Henrik Jacobsen. Residual flows for invertible generative modeling. Advances Neural Information Processing Systems, pages 9913-9923, 2019.
[44] Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning, pages 807-814, 2010.
[45] Weihao Xia, Yulun Zhang, Yujiu Yang. Jan Xue, Bolci Zhou, and Ming-Hsuan Yang. GAN inversion: A survey, arXiv preprint 2101::0:00000227. 2021.
[46] Nurit Spingarn, Ron Banner, and Tomer Michaeli. GAN "steerability" without optimization. In Interna- tional Conference on Learning Representations, 2021.
[47] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8026-8037, 2019.
[48] Luke Melas- Myriaz,,rrazz,,, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Finding unsupervised image segmenter in each of your deep generative models. arXiv preprint arXiv:2105.08112, 2021.
[49] Yann LeCun. Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-22333 2324, 1998.
[50] Pramnay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems, pages
2020.
[51] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 770-778. 2016.
[52] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[53] Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving contrastive learning by visualizing feature transformation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 00306-1031,, 2021.
[54] Ali Jahanian, Lucy Chai, and Phillip Isola. On the "steerability" of generative adversarial networks. In International Conference on Learning Representations, 2020.
[55] Tya Lashchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
[56] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. https: //gi //githun weriituu..co/aaae wwtp:///cccbooooeserreccccaaeeee dew.arrorrreeeeereneeea.ao 2019.
[57] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, pages 91 19,9 2015.
13

---

[58] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):003-338 2010.
[59] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision, pages 446-461, 2014.
[60] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Report, 2009.
[61] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages -392, 2010.
[62] Omkar Parkhi, Andrea Vedaldi, Andrew Zisserman, and Jawahar. Cats and dogs. In IEEE Conference on Computer Vixion and Puttern Recognition, pages 3498-3505, 2012.
[63] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual madels from few training examples: An incremental bayesian approach tested on 101 object categories. In Computer Vision and Pattern Recognition Workshop, pages 178-178, 2004.
[64] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729, 2008.
[65] Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. VISSL. https://githann tttpgttthuuuu... http///faceboooreeerrdcco
2021.
[66] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference on Computer Vision, pages 776-794, 2020.
[67] Colorado Reed, Sean Metzger, Aravind Srinivas, Trevor Darrell, and Kurt Keutzer. SelfAugment: leannin Automatic augmentation policies for self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2674-2683. 2021.
[68] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Revisiting contrastive methods for unsupervised learning of visual representations, In Advances Neural Information Processing Systems, pages 16238- 16250, 2021.
[69] Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. In Advances Neural Information Processing Systems, pages 15920-15933, 2021,
[70]
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukeuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach self- supervised learning. Advances in Neural Information Processing Systems, pages 21271-21284, 2020.
[71] Xinlei Chen and Kaiming He. Exploring simple siamese representation Icarning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[72] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems, pages 19667-19679, 2020.
[73] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, pages 8780-8794, 2021.
[74] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, FaannFrancois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. DataselGAN: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10145- 10155, 2021.
[75] Nontawat Tritrong, Pitchaporn Rewathoworwwong, and Supasorn Suwajanakorn. Repurposing GANs for one-shot semantic part segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4475-4485, 2021.
[76] Axel Sauer and Andreas Geiger. Counterfactual generative networks. In International Conference on Learning Representations, 2021.
[77] Lucy Chai, Jun- Jun-Yan Zhu, Eli Shechtman, Phillip Isola, and Richard Zhang. Ensembling with deep generative views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 149997-10007 2021.
14

---

[78] Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. In Advances in Neural Information Processing Systems, pages 12268-12279, 2019.
[79] Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez. This dataset does not exist: Training models from generated images. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1-5, 2020.
[80] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. GANSpace: Discovering in- terpretable GAN controls. In Advances in Neural Information Processing Systems, pages 9841-9850. 2020.
[81] Peiye Zhuang, Oluwasanmi Koyejo, and Alex Schwing. Enjoy your editing: Controllable GANs for image editing via latent space navigation. In International Conference on Learning Representations, 2021.
[82] Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in GANs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1532- 1540. 2021.
[83] Oguz Kaan Yüksel, Enis Simsar, Ezgi Gülperi Er, and Pinar Yanardag. LatentCLR: contrastive lcarning approach for unsupervised discovery of interpretable directions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14263-14272, 2021.
[84] Georgios A Kaissis, Marcus R Makowski, Daniel Rückert, and Rickmer F Braren. Secure, privacy- preserving and federated machine leaming in medical imaging. Nature Machine Intelligence, 2(6):305-311 2020.
[85] Yi Zeng, Enmeng Lu, Yinqian Sun, and Ruochen Tian. Responsible facial recognition and beyond. arXiv preprint arXiv: 12935, 2019.
[86] Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. Gan-leaks: A taxonomy of membership inference attacks against generative models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pages 443-362,222 2020.
[87] Muhammad Asim, Max Daniels, Oscar Leong, Ali Ahmed, and Paul Hand. Invertible generative models for inverse problems: mitigating representation error and dataset hias. In International Conference on Machine Learning, pages 499,409, 2020.
[88] Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. International Conference on Learning Representations, 2021.
[89] Ziyu Jiang, Tianlong Chen, Bohak Mortazavi, and Zhangyang Wang. Self-damaging contrastive learning. In International Conference on Machine Learning, pages 4927-4939, 2021.
[90] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pages (026-1034, 2015.
[91] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metrie. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018.
[92] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763, 2021.
[93] Igor Susmelj, Matthias Heller, Philipp Wirth, Jeremy Prescott, Malte Ebner, et al. Lightly. https: wttph//....oo/lggttyaaaaaa//a//a
2020.
[94] Arantxa Casanova, Marlene Careil, Jakob Verbeek, Michal Drozdzal. and Adriana Romero Soriano. Instanco-conditioned GAN. In Advances in Neural Information Processing Systems, pages 27517-27529, 2021.
[95] Tim Salimans, Ian Goodfellow, Wajciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems, 2016.
[96] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a time-scale update rule converge to local nash equilihrium. 2017.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Ys]] Please see the second paragraph in Section
(c) Did you discuss any potential negative socictal impacts of your work? [Yes] We discuss both positive and negative societal impacts of using generative models datasets in the last paragraph in Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] Please see Section 3.2.
(b) Did you include complete proofs of all theoretical results? [Yes] We give the full proof in Section 3.2 of the main text.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as URL)? [No] Our code and models will be released once this paper is accepted. We gave the pseudo code in Appendix A and experimental details in Section 4 and Appendix C.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see the detailed experimental settings in Section 4 and Appendix C.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] Because contrastive learning is computationally expensive, e.g., ~3 days for 100 epoch contrastive pretraining and ~1 day for the downstream object detection on 2 GPUs. It would be expensive for us to run all baselines and our method multiple times or perform longer pretraining, Nevertheless, consistent results can be observed from Section 4.3, where 200 epochs pretraining are conducted on smaller set of ImageNet.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see Appendix F.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the creators of datasets, pretrained models we use. Our codes are implemented based on GenRep, which together with the reproduced methods
mentioned in Appendix C.I.
(b) Did you mention the license of the assets? [Yes] Please Appendix C.I.
(c) Did you include any new assets either in the supplemental material or as URL? [N/A] We did not introduce new assets in this work.
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [Yes] We use public datasets (ImageNet, PASCAL VOC, and MNIST) and public pretrained generative models (BigBiGAN and BigGAN trained on ImageNet and Residual Flow trained on MNIST) which allow researchers to use.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] We use public datasets, as summarized above, contain objects, animals, numbers, etc.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "veTyHWJd4Xj", "reviewer": "Reviewer_58Jb", "summary": "This work proposes an approach to generate optimal positives for contrastive learning. The positive pairs are generated by finding an instance-specific latent space navigator $T_z$ by minimizing the mutual information between the positive pairs. \nThe synthetic data thus generated is used for self-supervised contrastive learning where the proposed approach outperforms the prior work on the Imagenet dataset and on the downstream task of object detection on the Pascal-VOC dataset.", "questions": "1. The choice of a good $\\delta$ in equation 5. The experiment section does not discuss the choice of $\\delta$ used in the paper. How is it chosen and what impact does training with different $\\delta$ have on the experiments in the paper. When increasing the number of samples in Figure 3 for training does a lower $\\delta$ mean redundant samples? It would be good to analyze the performance of the approach of different $\\delta$ and the different number of samples.\n\n2. In line 39: \"in contrast to representation space transformation and non-domain-knowledge-guided data augmentation\"; what kind of domain knowledge would be better preserved in this case?\n\n3. How the method compares to prior work in terms of computational efficiency (training time, resources) should be discussed in the main paper. Even though in the checklist, the method is claimed to be computationally expensive. The augmentation with the optimal positives would be useful to the community if it does not add too much computational overhead.\n\n4. The experimental results in table 1 are well within the standard deviation of the prior work. While its understandable that the experiments are time-consuming, the effect of different random seeds should be studied to establish the usefulness and significance of the results. \n", "limitations": "The paper adequately discusses the limitations of the approach and its dependence on pre-trained models. The discussion on how the biases are amplified with the proposed augmentation approach can be discussed in further detail. For example, what happens when increasing the number of positive pairs in case of a biased dataset?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper is easy to follow. The paper is well-written and motivated. The contributions of the work wrt to the prior work and finding instance-specific optimal positives for augmentation in self-supervised contrastive learning is clear. \n\n+ The experimental results outperform prior work trained on synthetic images.  The experimental results in line with prior work, demonstrate the proposed approach outperforms the prior approaches. The experiments are also performed on VOC object detection to show the applicability of the approach in downstream tasks.\n\n+ Ablations are thorough. The ablations conducted in Tables 2,3 and 4 show the benefits of the data augmentations as well as of the transformation applied in the latent space. \n\n- There are no major weaknesses, apart from the missing discussions on the choice of certain parameters (see Questions below) and comparison to prior work in terms of computational overhead. \n\n- The manuscript can be revised to fix the following typos: \n   *Equation 1, function sim not defined.\n   *Line 90: \"behind by formulating\" behind our approach?\n   *Line 44: \"computational\" to computationally.\n   *Line 239: \"as did in [60, 15, 28]\" to as done in.\n   \"Border Impact\" to Broader Impact\n  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nqlxz5HSPP2", "reviewer": "Reviewer_YrQG", "summary": "The work follows the GenRep paper and performs representation learning based on a generative model trained on data. They extends the GenRep by switching the fixed latent transformation to a learnable transform. They formulates COP-Gen as an instance-specific latent space navigator which minimizes the mutual information between the generated positive pair, and trains the encoder by maximizing the mutual information between positive pairs. It is clearly written and the theoretical part is correct. The idea is easy to follow and widely used in other frameworks such as GAN model. The performance is improved compared to the GenRep paper, but to be expected.", "questions": "- Why do you choose 224 as batch size? It might be better to align with other people's work and use 256 instead.\n\n- Could we combine this hard positive technique with non-contrastive learning frameworks, such as BYOL and SimSiam? Are there any experiment results on a small dataset, if the time is limited?\n\n- Why not use a Lagrange multiplier to contain the semantic distance between z and z'?\n\n- Normally training the model in a minimax way takes a longer time (epochs) to converge, or even struggles to converge. Could you provide a training/test loss plot with respect to the epochs?", "limitations": "The authors have adequately addressed the limitations and potential negative social impact of their work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- In previous works, people tries to improve the performance of contrastive learning by creating hard negative, while in this paper, the authors uses the InfoMin criterion to create optimal positives. The authors provided solid mathematical results to support the idea, and the proposed method is described and illustrated clearly.\n\nWeaknesses: \n- The performance is highly limited by the trained BigBiGAN model, although in theory with a perfect GAN model we could expect a comparable or even better results compared to SimCLR paper. \n\n- Minimax training objectives might struggles to converge to the optimal point, but this paper did not touch on this point. It would be helpful if more details about the training process is provided.\n\n- Some non-contrastive paper, such as BYOL and SimSiam should be included in references.\n\n- Minor point: \n  Typo: Border Impact ---> Broader Impact", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N0kUe79ABc-", "reviewer": "Reviewer_kg96", "summary": "This paper focuses on how to  generate Contrastive Optimal Positives (COP-Gen) for self-supervised contrastive learning and propose a new method based on pretrained generative models. Then, this paper presents theoretical analysis to illustrate that the proposed method can generate positives that removes as much nuisance information as possible while preserving the semantics. Experimental results show the effectiveness of the proposed method.", "questions": "1. A major problem faced by GANs is the mode collapse problem. This can be understood as learning only part of the semantic information. Therefore, using GAN for contrastive learning may lead to the problem of lack of semantics, so that it cannot better serve downstream tasks. How to solve this problem?\n2. I am very confused about Lemma 3.3. If Lemma 3.3. holds, then the learned semantic features are related to the input $Z$. However, $Z$ is a random noise and contains no semantic information.\n3. The caption of Figure 2 is puzzling.\n4. I hope the authors can provide the code as soon as possible. I want to verify this method in imagenet-1K dataset.\n5. How does the method perform on transfer tasks and semi-supervised learning tasks? I hope the author can give the corresponding experimental results.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is novelty, and the main idea is interesting.\n2. The experimental results is good.\n\nWeaknesses:\n1. The technical contribution is incremental.\n2. The theoretical analysis is vague. I think the learned transformations can not remove the task-independent semantic information only based on the Definition 2.1 and Proposition 3.1.\n3. The compared contrastive learning methods are outdated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N04u6Yto5ln", "reviewer": "Reviewer_yTbP", "summary": "This paper proposes a new framework -  Contrastive Optimal Positives Generation (COP-Gen) for self-supervised contrastive learning which aims to learn instance-specific latent transformations that minimizes the mutual information between the generated positive pair subject to the semantic consistency constraint.", "questions": "See weaknesses above", "limitations": "adequately addressed", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1.  The paper is well written and easy to follow.\n\n2. The idea of using a pre-trained generative network to obtain the positive samples is novel.\n\n### Weaknesses\nMy only concern about this paper is in the experimental part.\n\n1. I observed that the authors adopt a resolution of 112x112  images during the pertaining stage, and the reported performance for other methods is much lower than the numbers in the original papers. I think strictly following the setting from the original paper will be a fairer comparison. (e.g. 100 Epoch of SimCLR should reach 62.8% Top-1 Accuracy)\n\n2. Does the quality of the generative model affect the final result?\n\n3. What is the extra training cost if we include an additional generative model?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept