Contrastive Syn-to-Real Generalization
Authors: Wuyang Chen
Zhiding Yu
Shalini De Mello
Sifei Liu
Jose M. Alvarez
Zhangyang Wang
Anima Anandkumar
Venue: ICLR 2021 Poster (2021)
Keywords: synthetic-to-real generalization
domain generalization

Abstract: Training on synthetic data can be beneficial for label or data-scarce scenarios. However, synthetically trained models often suffer from poor generalization in real domains due to domain gaps. In this work, we make a key observation that the diversity of the learned feature embeddings plays an important role in the generalization performance. To this end, we propose contrastive synthetic-to-real generalization (CSG), a novel framework that leverage the pre-trained ImageNet knowledge to prevent overfitting to the synthetic domain, while promoting the diversity of feature embeddings as an inductive bias to improve generalization. In addition, we enhance the proposed CSG framework with attentional pooling (A-pool) to let the model focus on semantically important regions and further improve its generalization. We demonstrate the effectiveness of CSG on various synthetic training tasks, exhibiting state-of-the-art performance on zero-shot domain generalization.

Published as a conference paper at ICLR 2021
CONTRASTIVE SYN-TO-REAL GENERALIZATION
Wuyang Chen'r, Zhiding Yu Shalini De Mello", Sifei Liu?, Jose M. Alvarez?, Zhangyang Wang's Anima Anandkumar2.5 The University of Texas at Austin NVIDIA *California Institute of Technology
at attaaaaaaaangn atlaswang] edu [wuyang. chen, (zhidingy, shalinig, sifeil, josea, sanardkkuaz:@nnviddie https:/ //tp://ggtttb/////// www.eooooo//....... wom/WWabbs/CCC
ABSTRACT
Training on synthetic data can be beneficial for label or data-scarce scenarios. However, synthetically trained models often suffer from poor generalization in real domains due to domain gaps. In this work, we make key observation that the diver- sity of the learned feature embeddings plays an important role in the generalization performance. To this end, we propose contrastive wynthetic-to-ret generalization (CSG), a novel framework that leverages the pre- pre-trained ImageNet knowledge to prevent overfitting to the synthetic domain, while promoting the diversity of feature embeddings as an inductive bias to improve generalization. In addition, we enhance the proposed CSG framework with attentional pooling (A-pool) let the model focus on semantically important regions and further improve its generalization. We demonstrate the effectiveness of CSG on various synthetic training tasks, exhibiting state-of sitle-of-the-aae
the-ahr art performance on zero-shot domain generalization.
INTRODUCTION
Deep neural networks have pushed the boundaries of many visual recognition tasks, However, their success often hinges on the availability of both training data and labels. Obtaining data and labels can be difficult or expensive in many applications such as semantic segmentation, correspondence, 3D reconstruction, pose estimation, and reinforcement learning. In these cases, learning with synthetic data can greatly benefit the applications since large amounts of data and labels are available at relatively low costs. For this reason, synthetic training has recently gained significant attention (Wu et al., 2015; Richter et al., 2016; Shrivastava al., 2017; Savva et al., 2019).
Despite many benefits, synthetically trained models often have poor generalization on the real domain due to large domain gaps between synthetic and real images. Limi- tations simulation and rendering can lead degraded synthesis quality, such as aliased boundaries, unrealis- tic textures, fake appearance, over-simplified lighting conditions, and unreasonable scene layouts. These is- sues result in domain gaps between synthetic and real images, preventing the synthetically trained models from capturing meaningful representations and limiting their generalization ability on real images.
Labeled Training Bynthetic Data)
Unseen Test
[Real Data)
Pretrained Model egg. ImageNett
Visdo17
COCO
To mitigate these issues, domain generalization and adap- tation techniques have been proposed (Li et al., 2017; Pan al., 2018; Yue et al., 2019). Domain adaptation assumes the availability of target data (labeled, partially labeled, or unlabeled) during training. On the other hand, domain generalization considers zero-shot generalization without seeing the target data of real im- ages, and is therefore more challenging. An illustration of the domain generalization protecol on the
Figure 1: An illustration of the domain gen- eralization protocal on the VisDA-17 dataset, where real target domain (test) images are as- sumed unavailahle during model training.
WWor done during the research internship with NVIDIA. 1 'Corrsponning author.

---

Published as a conference paper at ICLR 2021
VisDA-17 dataset (Peng et al., 2017) is shown in Figure 1. Considering that ImageNet pre pre-trained representation is widely used as model initialization, recent efforts on domain generalization show that such knowledge be used prevent overfitting to the synthetic domain (Chen et al., 2018; 2020c). Specifically, they impose distillation loss to regularize the distance between the synthetically trained and the ImageNet pre-trained representations, which improves synthetic-to-real generalization.
The above approaches still face limitations due to the challenging nature of this problem. Taking a closer look, observe the following pitfalls in training on synthetic data. First, obtaining photo- realistic appearance features at the micro-level, such as texture and illumination, is challenging due to the limits of simulation complexity and rendering granularity. Without special treatment, CNNs tend to be biased towards textures (Geirhos et al., 2019) and suffer from badly learned representations on synthetic data. Second, the common lack of texture and shape variations on synthetic images often leads to collapsed and trivial representations without any diversity. This is unlike training with natural images where models get sufficiently trained by seeing enough variations. Such a lack of diversity in the representation makes the learned models vulnerable to natural variations in the real world.
Summary of contributions and results:
We observe that the diversity of learned feature embedding plays an important role in synthetic- to-real generalization. We show example of collapsed representations learned by synthetic model, which is in sharp contrast to features learned from real data (Section 2).
Motivated by the above observation, propose contrastive synthetic-to-real generalization framework that simultaneously regularizes the synthetically trained representation while promoting the diversity of the learned representation to improve generalization (Section 3.1).
We further enhance the CSG framework with attentional pooling (A-pool) where feature represen- tations guided by model attention. This allows the model localize its attention to semantically more important regions, and thus improves synthetic-to-red generalization (Section 3.4).
We benchmark CSG on various synthetic training tasks including image classification (VisDA-17) and semantic segmentation (GTAS > Cityscapes). We show that CSG considerably improves the generalization performance without seeing target data. Our best model reaches 64.05% accuracy on VisDA-17 compared to previous state-of-the-ee (Chen et al., 2020c) with 61.1% (Section 4).
A MOTIVATING EXAMPLE
We give a motivating example to show the significant differences between the features learned on synthetic and real images. Specifically, feature embedding after global average pooling (defined as O). We consider the following three models: 1) model pre-trained on ImageNet, 2) model trained on VisDA-17 validation set (real images), and 3) model trained on VisDA-17 training set (synthetic images) Both 2) and 3) are initialized with ImageNet pre-training, and fine-tuned on the 12 classes defined VisDA-17.
a ResNet-101 backbone and extract the la normalized
1.0
1.0
1.0
3.0
0.5
0.5
06
0.0
0.4
005
-0.5
-0.5
022
62
-10
-1.0
110
0.5
0.0
1.0
-1.0
-0.5
0.0
0.5
-1.0
-0,5 0.0
0.5 1.0
(a) ImageNet pre-trained (real).
(b) Trained on VisDA-17 validation
(c) Trained on VisDA-17 training set
(E, = 0.2541)
set (real), (E. 0.3355)
(synthetic). (E. 0.4408)
Figure 2: Feature diversity VisDA-17 test images B² with Gaussian kemne density estimation (KDE).
Darker areas more concentrated features. E,: hyperspherical energy features, lower the more diverse.
For a fair comparison, make random subset of the training set with equal size of the validation set, since the training set of VisDA-17 larger than the validation set.
2

---

Published as a conference paper at ICLR 2021
Visualization of feature diversity. We visualize the normalized representations 2-dim sphere. Gaussian kernel with bandwidth estimated by Scott's s Rule (Scott, 2015) is applied to estimate the probability density function. Darker areas have more concentrated features, and if the feature space (the 2-dim sphere) is covered by dark areas, it has more diversely placed features. In Figure 2, we see that the ImageNet pretrained model can widely span the representations on the 2-dim feature space. The model trained on VisDA-17 validation set can also generate diverse features, although slightly affected by the class imbalance. However, when the model is trained on the training set (synthetic images), the features largely collapse to a narrow subspace, i.e., the model fails to fully leverage the whole feature space. This clear that training on synthetic images can easily introduce poor bias to the model and the collapsed representations will fail to generalize to the real domain.
Quantitive measurement of feature diversity. Inspired by (Liu et al., 2018), we also quantitatively measure the diversity of the feature embeddings using the following hyperspherical potential energy:
N
1 Eis 16; új 0 8> 0 Eiz log ||0. 8, B,1l s=
E, D:
e, (1lē; -0y1)=
(1)
i=1
j=1,jş$
N is the number of examples. The lower the hyperspherical energy (HSE) is, the more diverse the feature vectors will be scattered in the unit sphere. 8 is the power factor, and we choose & = 0 in this example. Three training strategies exhibit energies as 0.2541, 0.3355, 0.4408, respectively. This validates that models trained on real images can capture diverse features, whereas the synthetic training will lead the model to highly collapsed feature space.
Remarks. A conclusion can be drawn from the above examples: though assisted with ImageNet initialization, fine-tuning on synthetic images tends to give collapsed features with poor diversity in sharp contrast to training with real images. This indicates that the diversity of learned representation could play an important role in synthetie-to-rr generalization.
CONTRASTIVE SYNTHETIC-TO-REAM GENERALIZATION
We consider the synthetic-to-reat -t--eell domain generalization problem following the protocols of Chen et al. (2020c). More specifically, the objective is to achieve the best zero-shot generalization on the unseen target domain real images without having access to them during synthetic training.
3.1 NOTATION AND FRAMEWORK
Our design of the model considers the following two aspects with "push and pull" strategy: Pull: Without access to real images, the ImageNet pre-trained model presents the only source real domain knowledge that implicitly guide our training. As result, we hope impose some form of similarity between the features obtained by the synthetic model and the ImageNet pre-trained one. This helps overcome the domain gaps from the unrealistic appearance of synthetic images.
Push: Section 2 shows that synthetic training tends to generate collapsed features whereas models trained on natural images give many diverse ones. We treat this as an inductive bias to improve synthetic training, by pushing the feature embeddings away from each other across different images.
The above "push and pull" strategy can be exactly formulated with a contrastive loss. This motivates us to propose contrastive synthetic-to-real generalization framework as partly inspired by recent popular contrastive learning methods (He al., 2020). Figure 3(b) illustrates our CSG framework. Specifically, we denote the frozen Imagenet pre- praited model as fc.. and the synthetically trained model fe, where fe is supervised by the task loss Exyn for the defined downstream task. We denote the input synthetic image as x and treat it as an anchor. We treat the embeddings of xt obtained by fe and fe.o as anchor and positive embeddings, denoting them as z" and x respectively. Following a typical contrastive approach, we define K negative images { x } for every anchor x", and denote their corresponding embeddings as {zT, .x)g Similar to the design in (Chen et al., 2020d), we define h/h : C + Re as the nonlinear projection heads with two MLP layers and a ReLU layer between them. The CSG framework regularizes fe in a contrastive manner: pulling z" and 2* to be closer while pushing z" and {zj .... 2 k } apart. This regularizes the model by preventing its representation from deviating too far from that of pre-trained ImageNet model and yet encouraging it to learn task-specific information from the synthetic data.
3

---

Published as a conference paper at ICLR 2021
fe.0
"X
fep
synthetic images
put
Feature Distance
synthetic images
x
pull
push
h
yxit
Sciee fe
fe
Contrastive
fo
Layn [Synthetic)
fe
L Syy (Synthetic)
(a)
(b)
Figure (a) Previous work (Chen al., 2018; 2020c) consider "learning without forgetting" which minimizes a distillation loss between a synthetic model and an ImageNet pre-trained one (cither on features or model parameters) to avoid catastrophic forgetting. (b) The proposed CSG framework with "push and pull" strategy.
Even though having connections to recent self-supervised contrastive representation learning meth- ods (Oord et al., 2018; Wu et al., 2018; Chen et al., 2020a; He et al., 2020; Chen et al., 2020b; Jiang work differs in the following aspects: 1) Self-supervised learning and the addressed
al., 2020), task are ill-posed in different manners the former lacks the constraints from semantic lahels, whereas the latter lacks the support of data distribution. 2) As result, the motivations of contrastive learning are different. Our work is also related to the contrastive distillation framework in (Tian et al., 2020a). Again, the two works differ in both task and motivation despite the converging techniques.
3.2 AUGMENTATION
Augmentation has been important part of effective contrastive learning. By perturbing providing different views of the representations, augmentation forces a model to focus more on the mid-level and high-level representations of object parts and structures which visually more realistic and reliable. To this end, we follow existing popular approaches to create augmentation at different levels:
Image augmentation. We consider image-level augmentation using RandAugment (Cubuk et al., 2020) where a single global control factor M is used to control the augmentation magnitude. We denote the transform operators of image-level augmentation T(.)-
Model augmentation. We adopt a mean-teate teacher (Tarvainen & Valpola, 2017) styled moving average of model create different views of feature embeddings. Given anchor image x" and K negative images { ... x k. we compute the embeddings as follows:
h =+ fe,o (T((²))) = og o h(T(z)))
where g Cx + F is pooling operator transforming feature map into vector. Follow- ing (He al., 2020), define hi(-) an exponential moving average of the h(-)) across different iterations. Such difference in fi(-) and hi(-) leads to augmented views embeddings.
3.3 CONTRASTIVE LOSS
We InfoNCE loss (Wu al., 2018) formulate the "push and pull" strategy:
exp (z" 2 x+/7)
LNCE -log
exp (z" exp 2 /T) E-- exp (2a((2"..) (2" /T)
where = 0.07 is temperature flyper-parametee in our work. Together, we minimize the combination of the synthetic task loss and LXCE during our transfer learning process:
Crask + ALNCE
(4)
Specifically, Crask is the synthetic training task objective. For example, Lrask is cross-entropy loss of a vector over the defined classes on VisDA-17, whereas a per-pixel dense cross-entropy loss GTA5. À is balancing factor controlling the strength of the Contrastive Learning.
4

---

Published as a conference paper at ICLR 2021
Multi-layer contrastive learning. We curious that which layer(s) should apply contrastive learning to achieve best generalization. We therefore propose a multi-layer CSG framework with different groups (combinations) of layer, denoted as g:
exp (z/a 2/: /T)
LNCE = EEkeR
Σ log LEG
(5)
exp (z 2by /T) + Σxi- exp (z! //)
LEG
We conduct ablation in Section 4.1.2 to study the generalization performance with respect to different G on ResNet-1012. Note that the non-linear projection heads 1 (.)))/ are layer-specifie.
Cross-task dense contrastive learning. Semantic segmentation presents a new form of task with per-pixel dense prediction, and the task naturally requires pixel-wise dense supervision Taskk Unlike image classification, an image in semantic segmentation could contain rich amounts of objects. We therefore make LNCE spatially denser in semantic segmentation make it more compatible with the dense task loss Task- Specifically, the NCE losses are applied on cropped feature map patches:
Ni LNCE ΣΣ LEG
NCE
ΣΣ
1 Ni
log
exp (2) la /T) E exp (z 2, /T)
exp
(6)
LEG am1
where we crop x into local patches ax with x = Se y h (TT(z))) Similar for x In practice, we crop x into N₁- = 8 x = 64 local patches during segmentation training.
3.4
A-POOL: ATTENTIONAL POOLING FOR IMPROVED REPRESENTATION
Σ Au
Moat Pooliig
(v)
fe
224
(a)
(b)
Figure 4: (a) For each input image, A A-pool computes attention matrix based the inner product between the global average pooled feature vector and vector at each position v.,.y (8, V.K. E R²). (b) Example of four generated reweighting matrices on different images. Note that the values are defined as the ratio of the attention over uniform weight. The attention is visualized with upsampling to match the input size (224 x224x
The purpose of the pooling function g(.) and the non-lineat projection head h(-) is to project high dimensional feature map v from Cxhxw to a low-dimensional embedding in Rs With the feature pooled by 9(-) being more informative, semantically meaningful representations. Inspired by recent works showing CNN's capability of localizing salient objects (Zhou al., 2016; Zhang al., 2018) with only image-level supervision, we propose an attentional pooling (A-pool) module to improve the quality of the pooled feature.
could also let the contrastive learning focus on more
As shown in Figure 4(a), given feature map first calculate its global average pooled vector E = g(v) = hew EE,, V1,ig. ... Enj vensll E [1, h1, E [1, w). we then define the attention score for each pixel at (i, as aijj (v (i' E [1, E [1, w)) and use this score as the weight term in global pooling. Specifically, we define A-pool operator as è = (v) [E, V1,6." ai.j. ... Ei, VC,i," a. This attention aitention-weighted weighted pooling procedure can effectively shift the focus of the pooled feature vector the semantically salient regions, leading to more meaningful contrastive learning. In Figure 4(b), we plot the attention as the ratio of new attention score a over uniform weights (i.e., the uniform score used in global average pooling xxw For example, a value 1 in Figure 4(b) indicates an attention score 1.5 related augmentation is applied, the attention used for fe,0 as described section 3.4 will be calculated by fer since fo is the one adapted to the source domain with better attention.
hxw Note that any spatially-
We follow the design by He et al. (2016) to group the convolution operators with the same input resolution layer which results four layer groups in ResNet-101.
5

---

Published as a conference paper at ICLR 2021
4
EXPERIMENT
We follow (Chen et al., 2020c) to evaluate on two popular benchmarks: VisDA-I7-COCO (classifi- cation) and GTA5 -iyyseeis (segmentation). Codes is available at https: /tt///gtttttt/// com/ NVlabs/ CSG.
4.1
IMAGE CLASSIFICATION
Dataset. The VisDA-17 dataset (Peng et al., 2017) provides three subsets (domains), each with the same 12 object categories. Among them, the training set (source domain) is collected from synthetic renderings 3D models under different angles and lighting conditions, whereas the validation set (target domain) contains real images cropped from the Microsoft COCO dataset (Lin al., 2014).
Implementation. For VisDA-17, we choose ImageNet pretrained ResNet- 101 (He et al., 2016) as the backbone. We fine-tune the model on the source domain with SGD optimizer of learning rate 1 x 10-4, weight decay 5 x 10-4, and momentum 0.9. Batch size is set to 32, and the model is trained for 30 epochs. for LNCE is set as 0.1.
4.1.1 MAIN RESULTS
We compare with different distillation strategies in Table 1, including feature l2 regularization (Chen et al., 2018), parameter /2 regularization, importance weighted parameter l2 regularization (Zenke al., 2017), and KI. divergence (Chen al., 2020c). All these approaches try retain the ImageNet domain knowledge during the synthetic training, without feature diversity being explicitly promoted. One could see, CSG significantly improves generalizaiton performance over these baselines.
We also verify that CSG promotes diverse representations, and that the diversity is correlated with generalization performance. To this end, we quantitatively measure the hyperspherical energy defined in Equation 1 on the feature embeddings extracted by different methods. From Table 1, one can see that the baseline suffers from the highest energy. and under different power settings, CSG consistently achieves the lowest energies. Table 1 indicates that method that achieves lower HSE can better generalize from synthetic to the real domain. This confirms our motivation that forcing the model to capture more diversely scattered features will achieve better generalization performance.
Table I: Generalization performance and hyperspherical energy of the features extracted by different models (lower better), Dataset: VisDA-17 (Peng et al., 2017) validation Model: ResNet-101.
Power
Model
Accuracy (%)
Oracle on ImageNet*
53.3
Baseline (vanilla synthetic training)
0,4245
1,2500
1.6028
49.3
Weight 12 distance (Kirkpatrick ct al., 2017)
0.4014
1.2296
1.5302
56.4
Synaptic Intelligence (Zenke al., 2017)
0.3958
1.2261
1.5216
57.6
Feature distance (Chen et al., 2018)
0.3337
1.1910
1.4449
57.1
ASG (Chen et al., 2020c)
0.3251
1.1840
1.4229
61.1
CSG (Ours)
0.3188
1.1806
1.4177
64.05
4.1.2 ABLATION STUDY
We perform ablation studies (Table 2, 4) on the VisDA-17 image classification benchmark (Peng et al., 2017).
Augmentation. We study different magnitudes RandAAgment (Cubuk et al., 2020) in our scenario (Section 3.2), as summarized in Table 2. By tuning the global magnitude control factor M, we observe that too strong augmentations deteriorate generalization (e.g. M 12, 18, 24), while mild augmentation brings limited help (M 3). A moderate augmentation (M = 6) can improve contrastive learning.
'The oracle is obtained by freezing the ResNet-101 classification layer on the VisDA-17 source domain (the FC layer for ImageNet remains unchanged). We use the PyTorch official model of prrtranetpretraineed ResNet-101.
101 backbone while only training the last new fully- colne-connected

---

Published as a conference paper at ICLR 2021
Input Image
Baseline
CSG (Ours)
Multi-layer Contrastive Learning. Since features from the high-level layers are di- rectly responsible for the downstream classi- fication or other vision tasks, we suspect the last layer in the feature extractor fe would be the most important. We conduct an ab- lation study on generalization performance with different layer combinations for multi- layer contrastive learning (Section 3.3). From Table 3, one can see that applying LNCE on layer and 4 are most effective. Therefore, in our work we set G = (3,4]
Airplane
Motorcydle
X
Blaycle
Figure 5: An illustration of model attention by GradCAM (Selvaraju et al., 2017) on the VisDA-17 validation set.
A-pool. Table shows that with attention guided pooling (Section 3.4), A-pool can further improve the generalization performance, compared with the vanilla global average pooling (GAP).
Table 2: Ablation with M.
Table 3: Ablation with G.
M
Accuracy
Table Ablation w./.oo A-peol.
Layer Groups G
Accuracy (%)
GAP: global average pooling.
0 (no aug.)
60.86
61.36
62.88
Pooling
Accuracy (%)
62.88
3+4
63.77
62.61
2+3+4
62.66
GAP A-pool
63.77
18
62.00
1+2+3+4
62.30
64.05
4.1.3
CSG BENEFITS VISUAL ATTENTION
We further show the Grad-CAM attention on VisDA-17 validation set (Figure 5). We can see that CSG framework also contributes to better visual attention on unseen real images.
4.2 SEMANTIC SEGMENTATION
Dataset. GTAS (Richter et al., 2016) a vehicle-egocentric image dataset collected in computer game with pixel-wise semantic labels. It contains 24,966 images with resolution of 1052x 1914. There are 19 classes that are compatible with the Cityscapes dataset (Cordts et al., 2016).
Cityscapes (Cordts et al., 2016) contains urban street images taken on vehicle from some European cities. There are 5,000 images with pixel-wise annotations. The images have a resolution of 1024x2048 and are labeled into 19 semantic categories.
Implementation. We study DeepLabv2 (Chen et al., 2017) with both ResNet-50 and ResNet- 101 backbone. The backbones are pre-trained ImageNet. We also use SGD) optimizer, with learning rate as 1 x 10-3, weight decay as x 10-4, and momentum are 0.9. Batch size is set to six. We crop the images into patches of 512x512 and train the model with multi-scale augmentation (0.75 1.25) and horizontal flipping. The model is trained for 50 epochs, and for LNCE is set as 75.
4.2.1 MAIN RESULTS
We also evaluate the generalization performance of our CSG semantic segmentation. In partieular, we treat the GTAS training set as the synthetic source domain and train segmentation models on it. We then treat the Cityscapes validation sets real target domains, where we directly evaluate the synthetically trained models. We can see that in Table 5, CSG achieves the best performance gain. IBN-Net Pan al. (2018) improves domain generalization by carefully mix the instance and batch normalization in the backbone, while Yue et al. (2019) transfers the real image styles from ImageNet to synthetic images. However, Yue et al. (2019) requires ImageNet images during synthetic training, and also implicitly leverages ImageNet labels as auxiliary domains.
"Grad-CAM visualization method (Selvaraju et al., 2017): https: / / thub. com/ ut kuczbul ak/ pytorch enn-vi sual izations
7

---

Published as a conference paper at ICLR 2021
Table 5: Comparison to previous domain generalization methods for segmentation (GTA5 Cityscapes).
Methods
Backbone
mloU %
mloU 1 %
No Adapt IBN-Net (Pan et al., 2018)
22.17
7.47
29.64
No Adapt Yue et al. (Yue et al., 2019)
32.45
4.97
ResNet-50
37.42
No Adapt ASG (Chen et al., 2020c)
25,88
3.77
29.65
No Adapt
25.88
9.39
CSG (ours)
35.27
No Adapt
33.56
8.97
Yue et al. (Yue et al., 2019)
42.53
No Adapt
ResNet-101
29.63
3.16
ASG (Chen et al., 2020c)
32.79
No Adapt
29.63
9.25
CSG (ours)
38,88
4.2.2 FEATURE DIVERSITY ON SEGMENTATION WITH BALANCED TRAINING SET
We further conduct visualization and quantitative measures of feature diversity on the segmentation task. Similar section we randomly sample subset of the GTAS training set to match the size of the Cityscapes training set. We again have similar observations: models trained on real images have relatively diverse features, and synthetic training leads to collapsed features. Here we get lower E, than classification since we follow the setting in Eq. 6 to study dense-level features. This leads to larger total number of features on segmentation than classification.
1.0
0.5
0.5
0.5
1.25
0.6
1.00
0.75
0.4
-0.5
110
-1.0
-1.0
-1.0
-0.5
0.0
0.5
1.0
-1.0 -0.5
0.0
0.5
1.0
-1.0
-05
0.0
1.0
(a) ImageNet pre-trained (real). (b) Trained on Cityscapes training (c) Trained on GTA5 training set
(E,
00.0888)
set (real). (E, -0.0643)
(synthetic). (E, = 0.0110)
Figure Feature diversity Cityscapes images in RR2 with Gaussian kemnel density estimation (KDE). Darker areas have more concentrated features. Eg: hyperspherical energy features, lower the more diverse.
4.2.3 VISUAL RESULTS
By visualizing the segmentation results (Figure 7), that CSG framework achieves better mloU on unseen real images from the Cityscapes validation set, the model produces segmen- tation with much higher visual quality. In contrast, the bascline model suffers from much more misclassification.
5 RELATED WORK
Domain generalization considers the problem of generalizing model to the unseen target domain without leveraging any target domain images (Muandet et al., 2013; Gan et al., 2016). The core challenge is how to close the domain gap and align feature spaces from different domains, without even seeing the target domain's data. Muandet al. (2013) proposed to use MMD (Maximum Mean Discrepancy) to align the distributions from different source domains and train their network with
8

---

Published as a conference paper at ICLR 2021
road Temaai
sidewalk sky
building
wall rider
Fence car
pole truck
Irathic lgt
trafhic
vegetation motoreycle
ignored
person
bus
Irain
bike
Figure 7: Generalization results on GTA5 - Cityscapes. Rows correspond to sample images in Cityscapes validation set. From left to right, columns correspond original images, ground truth, predication results of baseline SeepLabb22ReeNertie Chen al. (2017)), and prediction by model trained with our CSG framework.
adversarial learning. Li et al. (2017) built separate networks for each source domain and used shared parameters for testing. By using meta-learning approach on split training sets, Li al. (2018) further improved generalization performance. Instance Normalization and Batch Normalization are carefully integrated into the backbone network by Pan et al. (2018) to boost network generalization. Differently, Yue al. (2019) proposed to transfer information from the real domain image styles to synthetic images. Most recently, (Chen et al., 2020c) formulated domain generalization as a life-long learning problem (Li & Hoiem, 2017), and try avoid the catastrophic forgetting about the ImageNet pre-trained weights and to retain real-domain knowledge during transfer learning.
Contrastive learning. Noise contrastive estimation loss (Wu et al., 2018) recently becomes predominant design choice for sule-supe contrastive representation learning (Hjelm et al., 2018; Oord al., 2018; Hénaff al., 2019; Tian et al., 2019; He et al., 2020; Misra & Maaten, 2020; Chen et al., 2020a). Studies show that self-supervised models can serve as powerful initializations for downstream tasks, even outperforming supervised pre-training on several. Besides engineering improvements, key factors towards better contrastive learning include employing large numbers of negative examples and designing more semantically meaningful augmentations create different views of images. This leads to both maximize the mutual information between two views of the same instance and pushing examples from different instances apart (Tian et al., 2020b). As also observed by Wang & Isola (2020), contrastive learning tends to align the features belonging to the same instance, while scattering the normalized learned features on a hypersphere. However, most work focus on the representation learning for real-to-real transfer learning setting where the main focus is to improve the performance of the downstream tasks. While having connections to these methods, our work pursues different task with different motivations despite the converging techniques.
CONCLUSIONS
Motivated by the observation that models trained synthetic images tend to generate collapsed feature representation, we make a hypothesis that the diversity of feature representation plays an im- portant role in generalization performance. Taking this as inductive bias, we propose contrastive synthetic-to-real generalization framework that simultaneously regularizes the synthetically trained representations while promoting the diversity of the features to improve generalization. Experiments on VisDA-17 validate our hypothesis, showing that the diversity of features correlates with general- ization performance across different models. Together with the multi-scale contrastive learning and oveettionggidde pooling strategy, the proposed framework outperforms previous state-of- ontte-of-the-art on VisDA-17 with sizable gains, while giving competitive performance and the largest relative improvements on GTAS->Cityscapp without bells and whistles.

---

Published as a conference paper at ICLR 2021
REFERENCES
Liang-Chied Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous
convolution for semantic image segmentation. arXiv preprint arXiv: 005877 2017. 7,9
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05700 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020b.
Wuyang Chen, Zhiding Yu, Zhangyang Wang, and Animashree Anandkumar. Automated synthetic- to-real generalization. In International Conference on Machine Learning, pp. 1746-1756. PMLR, 2020c. 2, 3, 4, 6, 8,
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint WrXiv:203300229, 2020d.
Yuhua Chen, Wen Li, and Luc Van Gool, Road: Reality oriented adaptation for semantic segmentation of urban scenes. In CVPR, 2018. 2, 4, 6
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. CVPR Workshops, 2020.
Chuang Gan, Tianbao Yang, and Boqing Gong. Learning attributes equals multi-source domain generalization. In CVPR, 2016. 8
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In ICL.R, 2019. 2
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 5, 6
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 3, 4,9
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.09277 2019. 9
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchile Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. I Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808 066700 2018.
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. arXiv preprint arXiv:2010.13337. 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho. Agnieszka maabska-Barwinsk. et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526,, 2017. 6
Da Li,, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV, 2017. 1,
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. In AAAI, 2018.
10

---

Published as a conference paper at ICLR 2021
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. PAMI, 40(12)29355-2444 2017. 9
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. ECCV, 2014.
Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. NeurIPS, 2018.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of represtttii.......ppeettttttton In CVPR, 2020. 9
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICML, 2013.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint / 2018. 4,9
Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. In ECCV, 2018. 1,
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. VisDA: The visual domain adaptation challenge. arXiv preprint arXiv:1710.0622 2017. 2, 6
Stephan Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In ECCV, 2016. 7
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans. Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. ICCV, 2019. 1
David W Scott. Multivariate density estimation: theory, practice and visualization. John Wiley & Sons, 2015. 3
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2107-2116, 2017.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NewrIPS, 2017. 4
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint 19X.v: 2019. 9
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. NeurlPS, 2020b. 9
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. arXiv preprint arXiv:2005.10242, 2020. 9
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang. Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In CVPR, 2015.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018. 4,
11

---

Published as a conference paper at ICLR 2021
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni- Vincentelli, Kurt Keutzer, and Boqing Gong, Domain randomization and pyramid consistency: Simulation-to generalization without accessing target domain data. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2100-2110, 2019. 1, 7, 8, 9
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, 2017.
Xiaolin Zhang, Yunchao Wei, Guoliang Kang, Yi Yang, and Thomas Huang. Self Self-produced guidance for weakly-supervised object localization. In ECCV, 2018. 5
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
12


Reviews:
[{"review_id": "uliYWn8cEpO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Synthetic-to-real generalization is an important topic of extensive practical interest. This paper motivates its work from an observation of feature diversity difference between synthetic and real training: synthetic training tends to generate less diverse or even collapsed features, whereas models trained on natural images give much diverse ones. Based on that, the author developed a contrastive “push and pull” framework to: (1) keep proximity between learned and ImageNet-pretrained features; and (2) push the feature embeddings away from each other across different images, using the feature diversity observation as the inductive bias.  \n\nAlthough the proposed formulation looks similar to recent contrastive representation learning, this work seems to be well motivated, and find some unique interpretations for their push and pull in the specific context of syn-to-real generalization. The authors also proposed two variants: a multi-layer contrastive loss applied to intermediate feature map groups; and a cross-task dense contrastive loss that operates on the batch level for dense supervision.\n\nIt has been known that CNN tends to focus on textures and hence learns poorly from the less texture-rich synthetic data. It is further great to see the authors show an example illustrating the collapsed representations of a synthetic model, which clearly differs from those from training on real data. This provides a feature-level clear and measurable illustration of syn-real domain gap and implies the insight what good features should behave like in syn-to-real generalization.\n\nExperiments were on VisDA-17 (classification) and GTA5→Cityscapes (segmentation). Ablation studies were also presented on different augmentation and model components. They also proved again that the diversity of features correlates with generalization performance.\n\nWhile I feel overall positive, a number of questions and concerns remain for the authors:\n-\tIn Section 2 proof-of-concept, note that VisDA-17 training and validation sets have different sizes. Have you examined the potential confounding facor of different training steps, as longer training might be more prone to forgetting? I think a controlled experiment here could be training with two equal-sized subsets from VisDA-17 train/val.\n-\tSection 3.2: the authors seem to claim the two augmentations as their innovations, which may need be toned down. Using temporal averaged models inherits from MOCO, and image augmentation is now a standard in contrastive learning. \n-\tFurther, I wonder why the authors used RandAugment, instead of using the cascaded augmentation ways adopted by MOCO-V2/SimCLR. \n-\tSection 3.4: I am not sure if I fully get the point of A-pool. Is that the same as adding an attention layer in the nonlinear projection head?\n-\tWhere is the dense NCE loss applied? Was it adopted by both classification and segmentation experiments by default, or only in the segmentation? That was not specified anywhere in the paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "oHa3sD2uhfI", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper was motivated from an observation the common lack of texture and shape variations on synthetic images often leads the trained models to learning only collapsed and trivial representations without any diversity. The authors made a hypothesis that the diversity of feature representation would pay an important role in generalization performance and can be taken as an inductive bias. \n\nSeeing that, they proposed a synthetic-to-real generalization framework that simultaneously regularizes the synthetically trained representations while promoting the diversity of the features to improve generalization. Their strategy can be exactly formulated with a contrastive loss, which reminds me of Wang & Isola (2020) but was also customized for the synthetic-to-real generalization scenario. The framework was further enhanced by the multi-scale contrastive learning and an attention-guided pooling strategy. Besides, the dense contrastive loss (6) provided spatially denser patch-level supervision; that may be a novel idea that I haven’t seen before. However, the authors did not clarify where and how they use loss in their experiments.\n\nExperiments on VisDA-17 and GTA5 supported the hypothesis: though assisted with ImageNet initialization, fine-tuning on synthetic images tends to give collapsed features with poor diversity in sharp contrast to training with real images. This indicates that the diversity of learned representation could play an important role in synthetic-to-real generalization. Their experiments showed that the proposed framework can improve generalization by leveraging this inductive bias and can outperform previous state-of-the-arts without bells and whistles.\n\nI also feel more analysis and insights could have been provided for the segmentation experiments in 4.2. Currently there is no more information beyond Table 5. For example, some feature diversity measure like Table 1 could be reported for segmentation too, since revealing the feature diversity inductive bias is the main novelty in this paper. Also, more clarifications are needed on comparing the settings fairly with prior work like Pan et al. (2018) and Yue et al. (2019).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "m4IbVV1-2jV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Training on synthetic data and generalizing to real test data is an important task that can be particularly beneficial for label or data-scarce scenarios. The paper aims to achieve the best zero-shot generalization on the unseen target domain real images without having access to them during synthetic training. The experiments thoughtfully demonstrate both classification and segmentation, as well as ablation studies. Visualizations and interpretations are presented in addition. The visual interpretation of feature diversity in Sec. 2 is another plus. Overall this paper is good both conceptually and experimentally. It is also well written in general. \n\nMy main concern is that the current baseline comparisons in experiment are not fully consistent nor satisfactory. As observed from Table 5, by absolute numbers (not relative margin), the proposed method is roughly aligned with ASG and IBN-Net, but lags much behind (Yue et al., 2019). That was partially explained by the authors, by saying that Yue et al. (2019) required ImageNet images during synthetic training and also implicitly leveraged ImageNet labels as auxiliary domains. However, if looking more closely, even the baseline ResNet-50 mIoU sees a big gap between Yue et al. (2019), and the proposed method as well as the two others. It is unclear and unconvincing to me why the same baseline can perform so differently among those methods, and I think this might potentially undermine the experiment reliability/reproducibility and deserves more clarification from the authors. \n\nOne more nitpick is that this paper shows no figure in experiments. Given there is some extra space, the authors may want to visualize some classification and segmentation results, displaying both success and failure cases.\n\nTypos:\n\na novel framework that leverage  -> should be “leverages”\n\nthe diversity of learned feature embedding play -> should be “plays”\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fgpu9KbN7d", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper focuses on the domain generalization problem where the source domain contains synthetic data. An interesting phenomenon is observed in this paper: the diversity of the learned feature embeddings plays an important role in the generalization performance. Then, this paper presents a method to address the syn-to-real generalization problem by combining augmentation, contrastive loss and attention pooling techniques. In general, the observed phenomenon is interesting but the proposed method is not a well-motivated solution. Besides, the researched problem is not a general one, which limits the impact of this paper.\n\nPros:\n\n1. When the source domain only contains synthetic data, this paper observes that the diversity of the learned feature embeddings plays an important role in the generalization performance in Section 2. From Figure 2, it is clear that the synthetic data are very different from real data in the view of feature diversity. This phenomenon is interesting and maybe motivate more works regarding syn-to-real problems (not only the domain generalization problem).\n\nCons:\n\n1. This paper provides a new view of addressing the syn-to-real generalization problem. However, the solution presented in this paper is not novel and is not connected with the new view very well. It is not novel enough by only combining augmentation, contrastive loss and attention pooling techniques.\n\n2. The experiments are not enough. For general interest, this paper should also present results when using the proposed method to address ordinary domain generalization problems. The researched problem is not a general one, which limits the impact of this paper.\n\n3. There are many typos in this paper (even in the abstract: \"that leverage\" should be \"that leverages\").\n\n4. It is unclear why we should use augmentation and A-pool. The motivations behind the two techniques are unclear. Why can they improve the accuracy of the proposed method?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper raised a natural question: why good synthetic images can be not so good at training/fine-tuning models for downstream tasks (e.g., classification and segmentation)? This problem is named synthetic-to-real (domain) generalization (where syn/real images are regarded as from the source/target domain), and it is of practical importance when using GAN-like methods given limited real images for training. The authors found that the answer to the question is the diversity of the learned feature embeddings, and argued/advocated that we should encourage such diversity when training on syn images in order to better approximate training on real images. To this end, a novel contrastive synthetic-to-real generalization framework was proposed and shown effective in the well designed experiments.

Overall, the quality is above the bar. While some reviewers had some concerns about the applicability and the motivations for the algorithm design, the authors have done a particularly good job in the rebuttal. After the rebuttal, we all think the paper should be accepted for publication.

I have some comments on the writing. The introduction claiming so many things has only 4 citations, especially the first two paragraphs have no citation. While I do think what claimed there are correct, the authors should include certain supportive evidences after each claim by themselves. Moreover, while I do think the problem hunting part is well motivated, the problem solving part needs its own motivation/justification. When two or more components are combined in a proposal, why this component is chosen and is there other choice that can achieve the same purpose (this concern has also been raised by reviewers)? I believe the components are not randomly chosen among possible candidates (e.g., "we further enhance the CSG framework with attentional pooling"), but for writing a paper, the authors should explain the motivation for the algorithm design because we cannot know the motivation unless they tell us.
Decision: Accept (Poster)