Contrastive Syn-to-Real Generalization

1 INTRODUCTION . Deep neural networks have pushed the boundaries of many visual recognition tasks . However , their success often hinges on the availability of both training data and labels . Obtaining data and labels can be difficult or expensive in many applications such as semantic segmentation , correspondence , 3D reconstruction , pose estimation , and reinforcement learning . In these cases , learning with synthetic data can greatly benefit the applications since large amounts of data and labels are available at relatively low costs . For this reason , synthetic training has recently gained significant attention ( Wu et al. , 2015 ; Richter et al. , 2016 ; Shrivastava et al. , 2017 ; Savva et al. , 2019 ) . Despite many benefits , synthetically trained models often have poor generalization on the real domain due to large domain gaps between synthetic and real images . Limitations on simulation and rendering can lead to degraded synthesis quality , such as aliased boundaries , unrealistic textures , fake appearance , over-simplified lighting conditions , and unreasonable scene layouts . These issues result in domain gaps between synthetic and real images , preventing the synthetically trained models from capturing meaningful representations and limiting their generalization ability on real images . To mitigate these issues , domain generalization and adaptation techniques have been proposed ( Li et al. , 2017 ; Pan et al. , 2018 ; Yue et al. , 2019 ) . Domain adaptation assumes the availability of target data ( labeled , partially labeled , or unlabeled ) during training . On the other hand , domain generalization considers zero-shot generalization without seeing the target data of real images , and is therefore more challenging . An illustration of the domain generalization protocol on the ∗Work done during the research internship with NVIDIA . †Corresponding author . VisDA-17 dataset ( Peng et al. , 2017 ) is shown in Figure 1 . Considering that ImageNet pre-trained representation is widely used as model initialization , recent efforts on domain generalization show that such knowledge can be used to prevent overfitting to the synthetic domain ( Chen et al. , 2018 ; 2020c ) . Specifically , they impose a distillation loss to regularize the distance between the synthetically trained and the ImageNet pre-trained representations , which improves synthetic-to-real generalization . The above approaches still face limitations due to the challenging nature of this problem . Taking a closer look , we observe the following pitfalls in training on synthetic data . First , obtaining photorealistic appearance features at the micro-level , such as texture and illumination , is challenging due to the limits of simulation complexity and rendering granularity . Without special treatment , CNNs tend to be biased towards textures ( Geirhos et al. , 2019 ) and suffer from badly learned representations on synthetic data . Second , the common lack of texture and shape variations on synthetic images often leads to collapsed and trivial representations without any diversity . This is unlike training with natural images where models get sufficiently trained by seeing enough variations . Such a lack of diversity in the representation makes the learned models vulnerable to natural variations in the real world . Summary of contributions and results : • We observe that the diversity of learned feature embedding plays an important role in syntheticto-real generalization . We show an example of collapsed representations learned by a synthetic model , which is in sharp contrast to features learned from real data ( Section 2 ) . • Motivated by the above observation , we propose a contrastive synthetic-to-real generalization framework that simultaneously regularizes the synthetically trained representation while promoting the diversity of the learned representation to improve generalization ( Section 3.1 ) . • We further enhance the CSG framework with attentional pooling ( A-pool ) where feature representations are guided by model attention . This allows the model to localize its attention to semantically more important regions , and thus improves synthetic-to-real generalization ( Section 3.4 ) . • We benchmark CSG on various synthetic training tasks including image classification ( VisDA-17 ) and semantic segmentation ( GTA5→ Cityscapes ) . We show that CSG considerably improves the generalization performance without seeing target data . Our best model reaches 64.05 % accuracy on VisDA-17 compared to previous state-of-the-art ( Chen et al. , 2020c ) with 61.1 % ( Section 4 ) . 2 A MOTIVATING EXAMPLE . We give a motivating example to show the significant differences between the features learned on synthetic and real images . Specifically , we use a ResNet-101 backbone and extract the l2 normalized feature embedding after global average pooling ( defined as v̄ ) . We consider the following three models : 1 ) model pre-trained on ImageNet , 2 ) model trained on VisDA-17 validation set ( real images ) , and 3 ) model trained on VisDA-17 training set ( synthetic images ) 1 . Both 2 ) and 3 ) are initialized with ImageNet pre-training , and fine-tuned on the 12 classes defined in VisDA-17 . 1.0 0.5 0.0 0.5 1.0 1.0 0.5 0.0 0.5 1.0 0.0 0.2 0.4 0.6 0.8 1.0 ( a ) ImageNet pre-trained ( real ) . ( Es = 0.2541 ) 1.0 0.5 0.0 0.5 1.0 1.0 0.5 0.0 0.5 1.0 0.0 0.2 0.4 0.6 0.8 1.0 ( b ) Trained on VisDA-17 validation set ( real ) . ( Es = 0.3355 ) 1.0 0.5 0.0 0.5 1.0 1.0 0.5 0.0 0.5 1.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 ( c ) Trained on VisDA-17 training set ( synthetic ) . ( Es = 0.4408 ) Visualization of feature diversity . We visualize the normalized representations on a 2-dim sphere . A Gaussian kernel with bandwidth estimated by Scott ’ s Rule ( Scott , 2015 ) is applied to estimate the probability density function . Darker areas have more concentrated features , and if the feature space ( the 2-dim sphere ) is covered by dark areas , it has more diversely placed features . In Figure 2 , we can see that the ImageNet pretrained model can widely span the representations on the 2-dim feature space . The model trained on VisDA-17 validation set can also generate diverse features , although slightly affected by the class imbalance . However , when the model is trained on the training set ( synthetic images ) , the features largely collapse to a narrow subspace , i.e. , the model fails to fully leverage the whole feature space . This is clear that training on synthetic images can easily introduce poor bias to the model and the collapsed representations will fail to generalize to the real domain . Quantitive measurement of feature diversity . Inspired by ( Liu et al. , 2018 ) , we also quantitatively measure the diversity of the feature embeddings using the following hyperspherical potential energy : Es ( v̄i|Ni=1 ) = N∑ i=1 N∑ j=1 , j 6=i es ( ‖v̄i − v̄j‖ ) = { ∑ i6=j ‖v̄i − v̄j‖ −s , s > 0∑ i6=j log ( ‖v̄i − v̄j‖−1 ) , s = 0 ( 1 ) N is the number of examples . The lower the hyperspherical energy ( HSE ) is , the more diverse the feature vectors will be scattered in the unit sphere . s is the power factor , and we choose s = 0 in this example . Three training strategies exhibit energies as 0.2541 , 0.3355 , 0.4408 , respectively . This validates that models trained on real images can capture diverse features , whereas the synthetic training will lead the model to highly collapsed feature space . Remarks . A conclusion can be drawn from the above examples : though assisted with ImageNet initialization , fine-tuning on synthetic images tends to give collapsed features with poor diversity in sharp contrast to training with real images . This indicates that the diversity of learned representation could play an important role in synthetic-to-real generalization . 3 CONTRASTIVE SYNTHETIC-TO-REAL GENERALIZATION . We consider the synthetic-to-real domain generalization problem following the protocols of Chen et al . ( 2020c ) . More specifically , the objective is to achieve the best zero-shot generalization on the unseen target domain real images without having access to them during synthetic training . 3.1 NOTATION AND FRAMEWORK . Our design of the model considers the following two aspects with a “ push and pull ” strategy : Pull : Without access to real images , the ImageNet pre-trained model presents the only source of real domain knowledge that can implicitly guide our training . As a result , we hope to impose some form of similarity between the features obtained by the synthetic model and the ImageNet pre-trained one . This helps to overcome the domain gaps from the unrealistic appearance of synthetic images . Push : Section 2 shows that synthetic training tends to generate collapsed features whereas models trained on natural images give many diverse ones . We treat this as an inductive bias to improve synthetic training , by pushing the feature embeddings away from each other across different images . The above “ push and pull ” strategy can be exactly formulated with a contrastive loss . This motivates us to propose a contrastive synthetic-to-real generalization framework as partly inspired by recent popular contrastive learning methods ( He et al. , 2020 ) . Figure 3 ( b ) illustrates our CSG framework . Specifically , we denote the frozen Imagenet pre-trained model as fe , o and the synthetically trained model fe , where fe is supervised by the task loss Lsyn for the defined downstream task . We denote the input synthetic image as xa and treat it as an anchor . We treat the embeddings of xa obtained by fe and fe , o as anchor and positive embeddings , denoting them as za and z+ , respectively . Following a typical contrastive approach , we define K negative images { x−1 , · · · , x − K } for every anchor xa , and denote their corresponding embeddings as { z−1 , · · · , z − K } . Similar to the design in ( Chen et al. , 2020d ) , we define h/h̃ : RC → Rc as the nonlinear projection heads with a two MLP layers and a ReLU layer between them . The CSG framework regularizes fe in a contrastive manner : pulling za and z+ to be closer while pushing za and { z−1 , · · · , z − K } apart . This regularizes the model by preventing its representation from deviating too far from that of a pre-trained ImageNet model and yet encouraging it to learn task-specific information from the synthetic data . 𝑓 '' 𝑓 # 𝑓 '' , % ℒ ' ( ) 𝒙+ frozen updated ( Synthetic ) synthetic images Feature Distance pull 𝒗′ 𝒗 𝑓 '' 𝑓 # 𝑓 '' , % ℒ ' ( ) 𝒙 frozen updated ( Synthetic ) synthetic images Feature Distance pull ( a ) 𝑓 '' 𝑓 # 𝑓 '' , % ℒ ' ( ) 𝒙𝒂 frozen updated ( Synthetic ) synthetic images 𝒛- , . 𝒙/0 , ⋯ , 𝒙30 𝒛/ -,0 , ⋯ , 𝒛3 -,0𝒛-,4 pushpull ; ℎ 𝒗4 𝒗𝒂 𝒗0 𝒗 ℎ Contrastive ( b ) Figure 3 : ( a ) Previous work ( Chen et al. , 2018 ; 2020c ) consider “ learning without forgetting ” which minimizes a distillation loss between a synthetic model and an ImageNet pre-trained one ( either on features or model parameters ) to avoid catastrophic forgetting . ( b ) The proposed CSG framework with a “ push and pull ” strategy . Even though having connections to recent self-supervised contrastive representation learning methods ( Oord et al. , 2018 ; Wu et al. , 2018 ; Chen et al. , 2020a ; He et al. , 2020 ; Chen et al. , 2020b ; Jiang et al. , 2020 ) , our work differs in the following aspects : 1 ) Self-supervised learning and the addressed task are ill-posed in different manners - the former lacks the constraints from semantic labels , whereas the latter lacks the support of data distribution . 2 ) As a result , the motivations of contrastive learning are different . Our work is also related to the contrastive distillation framework in ( Tian et al. , 2020a ) . Again , the two works differ in both task and motivation despite the converging techniques .

Summary: Synthetic-to-real generalization is an important topic of extensive practical interest. This paper motivates its work from an observation of feature diversity difference between synthetic and real training: synthetic training tends to generate less diverse or even collapsed features, whereas models trained on natural images give much diverse ones. Based on that, the author developed a contrastive “push and pull” framework to: (1) keep proximity between learned and ImageNet-pretrained features; and (2) push the feature embeddings away from each other across different images, using the feature diversity observation as the inductive bias.