Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper was motivated from an observation the common lack of texture and shape variations on synthetic images often leads the trained models to learning only collapsed and trivial representations without any diversity. The authors made a hypothesis that the diversity of feature representation would pay an important role in generalization performance and can be taken as an inductive bias. 

Seeing that, they proposed a synthetic-to-real generalization framework that simultaneously regularizes the synthetically trained representations while promoting the diversity of the features to improve generalization. Their strategy can be exactly formulated with a contrastive loss, which reminds me of Wang & Isola (2020) but was also customized for the synthetic-to-real generalization scenario. The framework was further enhanced by the multi-scale contrastive learning and an attention-guided pooling strategy. Besides, the dense contrastive loss (6) provided spatially denser patch-level supervision; that may be a novel idea that I haven’t seen before. However, the authors did not clarify where and how they use loss in their experiments.

Experiments on VisDA-17 and GTA5 supported the hypothesis: though assisted with ImageNet initialization, fine-tuning on synthetic images tends to give collapsed features with poor diversity in sharp contrast to training with real images. This indicates that the diversity of learned representation could play an important role in synthetic-to-real generalization. Their experiments showed that the proposed framework can improve generalization by leveraging this inductive bias and can outperform previous state-of-the-arts without bells and whistles.

I also feel more analysis and insights could have been provided for the segmentation experiments in 4.2. Currently there is no more information beyond Table 5. For example, some feature diversity measure like Table 1 could be reported for segmentation too, since revealing the feature diversity inductive bias is the main novelty in this paper. Also, more clarifications are needed on comparing the settings fairly with prior work like Pan et al. (2018) and Yue et al. (2019).

review_title: idea is novel, more technical details needed

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

This paper focuses on the domain generalization problem where the source domain contains synthetic data. An interesting phenomenon is observed in this paper: the diversity of the learned feature embeddings plays an important role in the generalization performance. Then, this paper presents a method to address the syn-to-real generalization problem by combining augmentation, contrastive loss and attention pooling techniques. In general, the observed phenomenon is interesting but the proposed method is not a well-motivated solution. Besides, the researched problem is not a general one, which limits the impact of this paper.

Pros:

1. When the source domain only contains synthetic data, this paper observes that the diversity of the learned feature embeddings plays an important role in the generalization performance in Section 2. From Figure 2, it is clear that the synthetic data are very different from real data in the view of feature diversity. This phenomenon is interesting and maybe motivate more works regarding syn-to-real problems (not only the domain generalization problem).

Cons:

1. This paper provides a new view of addressing the syn-to-real generalization problem. However, the solution presented in this paper is not novel and is not connected with the new view very well. It is not novel enough by only combining augmentation, contrastive loss and attention pooling techniques.

2. The experiments are not enough. For general interest, this paper should also present results when using the proposed method to address ordinary domain generalization problems. The researched problem is not a general one, which limits the impact of this paper.

3. There are many typos in this paper (even in the abstract: "that leverage" should be "that leverages").

4. It is unclear why we should use augmentation and A-pool. The motivations behind the two techniques are unclear. Why can they improve the accuracy of the proposed method?
review_title: The observed phenomenon is interesting but the proposed method is not a well-motivated solution.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Training on synthetic data and generalizing to real test data is an important task that can be particularly beneficial for label or data-scarce scenarios. The paper aims to achieve the best zero-shot generalization on the unseen target domain real images without having access to them during synthetic training. The experiments thoughtfully demonstrate both classification and segmentation, as well as ablation studies. Visualizations and interpretations are presented in addition. The visual interpretation of feature diversity in Sec. 2 is another plus. Overall this paper is good both conceptually and experimentally. It is also well written in general. 

My main concern is that the current baseline comparisons in experiment are not fully consistent nor satisfactory. As observed from Table 5, by absolute numbers (not relative margin), the proposed method is roughly aligned with ASG and IBN-Net, but lags much behind (Yue et al., 2019). That was partially explained by the authors, by saying that Yue et al. (2019) required ImageNet images during synthetic training and also implicitly leveraged ImageNet labels as auxiliary domains. However, if looking more closely, even the baseline ResNet-50 mIoU sees a big gap between Yue et al. (2019), and the proposed method as well as the two others. It is unclear and unconvincing to me why the same baseline can perform so differently among those methods, and I think this might potentially undermine the experiment reliability/reproducibility and deserves more clarification from the authors. 

One more nitpick is that this paper shows no figure in experiments. Given there is some extra space, the authors may want to visualize some classification and segmentation results, displaying both success and failure cases.

Typos:

a novel framework that leverage  -> should be “leverages”

the diversity of learned feature embedding play -> should be “plays”

review_title: Official Reivew of "Contrastive Syn-to-Real Generalization"

final_score: rating: 7: Good paper, accept
review_content: Synthetic-to-real generalization is an important topic of extensive practical interest. This paper motivates its work from an observation of feature diversity difference between synthetic and real training: synthetic training tends to generate less diverse or even collapsed features, whereas models trained on natural images give much diverse ones. Based on that, the author developed a contrastive “push and pull” framework to: (1) keep proximity between learned and ImageNet-pretrained features; and (2) push the feature embeddings away from each other across different images, using the feature diversity observation as the inductive bias.  

Although the proposed formulation looks similar to recent contrastive representation learning, this work seems to be well motivated, and find some unique interpretations for their push and pull in the specific context of syn-to-real generalization. The authors also proposed two variants: a multi-layer contrastive loss applied to intermediate feature map groups; and a cross-task dense contrastive loss that operates on the batch level for dense supervision.

It has been known that CNN tends to focus on textures and hence learns poorly from the less texture-rich synthetic data. It is further great to see the authors show an example illustrating the collapsed representations of a synthetic model, which clearly differs from those from training on real data. This provides a feature-level clear and measurable illustration of syn-real domain gap and implies the insight what good features should behave like in syn-to-real generalization.

Experiments were on VisDA-17 (classification) and GTA5→Cityscapes (segmentation). Ablation studies were also presented on different augmentation and model components. They also proved again that the diversity of features correlates with generalization performance.

While I feel overall positive, a number of questions and concerns remain for the authors:
-	In Section 2 proof-of-concept, note that VisDA-17 training and validation sets have different sizes. Have you examined the potential confounding facor of different training steps, as longer training might be more prone to forgetting? I think a controlled experiment here could be training with two equal-sized subsets from VisDA-17 train/val.
-	Section 3.2: the authors seem to claim the two augmentations as their innovations, which may need be toned down. Using temporal averaged models inherits from MOCO, and image augmentation is now a standard in contrastive learning. 
-	Further, I wonder why the authors used RandAugment, instead of using the cascaded augmentation ways adopted by MOCO-V2/SimCLR. 
-	Section 3.4: I am not sure if I fully get the point of A-pool. Is that the same as adding an attention layer in the nonlinear projection head?
-	Where is the dense NCE loss applied? Was it adopted by both classification and segmentation experiments by default, or only in the segmentation? That was not specified anywhere in the paper.

review_title: Accept

Meta-review: The paper raised a natural question: why good synthetic images can be not so good at training/fine-tuning models for downstream tasks (e.g., classification and segmentation)? This problem is named synthetic-to-real (domain) generalization (where syn/real images are regarded as from the source/target domain), and it is of practical importance when using GAN-like methods given limited real images for training. The authors found that the answer to the question is the diversity of the learned feature embeddings, and argued/advocated that we should encourage such diversity when training on syn images in order to better approximate training on real images. To this end, a novel contrastive synthetic-to-real generalization framework was proposed and shown effective in the well designed experiments.

Overall, the quality is above the bar. While some reviewers had some concerns about the applicability and the motivations for the algorithm design, the authors have done a particularly good job in the rebuttal. After the rebuttal, we all think the paper should be accepted for publication.

I have some comments on the writing. The introduction claiming so many things has only 4 citations, especially the first two paragraphs have no citation. While I do think what claimed there are correct, the authors should include certain supportive evidences after each claim by themselves. Moreover, while I do think the problem hunting part is well motivated, the problem solving part needs its own motivation/justification. When two or more components are combined in a proposal, why this component is chosen and is there other choice that can achieve the same purpose (this concern has also been raised by reviewers)? I believe the components are not randomly chosen among possible candidates (e.g., "we further enhance the CSG framework with attentional pooling"), but for writing a paper, the authors should explain the motivation for the algorithm design because we cannot know the motivation unless they tell us.
Decision: Accept (Poster)