Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary:  This paper proposed adversarial teacher-student knowledge distillation for domain generalization. Motivated by recent progress in self-supervised representation learning, the authors proposed two learning stages including domain generalization representation learning and novel domain augmentation. During domain generalized representation learning, the student is updated while the augmenter is fixed and the loss minimizes the discrepancy between the teacher and student. During the Novel domain augmentation learning, the student is fixed and the augmenter is updated. The objective is to maximize the discrepancy and encourage the augmenter to generate more distinct samples. Experiments on several public benchmark datasets very the effectiveness of the proposed method. 
main_review: [Strength]

The paper is well written and the idea is novel. The authors perform extensive experiments and ablation studies to verify the effectiveness of each model.  The proposed 2 stage representation and augmentation learning dynamically combine cycle GAN and self-supervised learning by knowledge distillation. The insight that the teacher module is avoided from observing the unrealistic domain augmentation is interesting and verified from experiment results. The results are very good and achieve state-of-the-art performance on multiple benchmarks.  

[Weakness]

In the Siamese archi. (From table 5), the paper shows optimizing both teacher and student through gradient descents still achieve reasonable performance. In self-supervised image representation learning, there will be severe mode collapses without EMA update. I wonder why siamese architecture is working here? Could the author illustrate more. 
limitations_and_societal_impact: N.A.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies the problem of Domain Generalization (DG), where a model trained on single or multiple source distributions / domains is expected to generalize to data from unseen / novel distributions at test-time. The authors propose an adversarial teacher-student representation learning setup consisting of two stages -- (1) synthesizing “plausible” out-of-domain instances as augmented source data (termed Novel Domain Augmentation) and (2) enforcing the representations obtained from augmented and vanilla source domain instances to be similar and relevant to the task at hand (termed Domain Generalized Representation Learning). This is achieved via teacher (fed regular instances) and student (fed augmented instance) encoders and a novel data augmentor unit. In the second stage, the student encoder is incentivized to be good at the downstream task and ensure representations from augmented samples are similar to the vanilla ones (the teacher parameters are kept up to date via an EMA step). To generate novel augmented samples, the augmentor unit is incentivized to maximize the discrepancy between the representations obtained from the teacher and student encoders. Broadly, the proposed approach falls in the category of DG relying on hallucinating reasonable out-of-domain samples (without assuming any access to domain labels) during training. Obtained results indicate that the proposed approach is competitive with existing DG approaches.
main_review: I will now highlight the strengths and weaknesses of the submission, which will address quality, clarity, originality and significance of the submission.

With the exception of the points covered under weaknesses, I believe the following are the strengths of the submission in its current form:

- The paper is generally well-written and easy to follow. With the exception of a few points under weaknesses regarding clarity issues, the authors do a decent job of motivating several components of the proposed approach -- the multi-view perspective on Domain Generalization, the Adversarial Teacher Student setup and other design choices.

- From a modelling standpoint, perhaps one of the biggest strengths of the paper lies in the fact that it does not rely on domain-labels -- making it useful for not only the multi-source but also the single-source domain generalization (DG) setup. More importantly, there is value on relying on DG approaches agnostic to domain labels especially for settings where access to domain labels may be a restrictive assumption (due to annotation cost / lack of clarity in terms of what constitutes different domains -- for instance, datasets like CARLA, CityScapes, etc.).

- The proposed approach seems simple, intuitive and results in competitive (on PACS) and improved (on Office-Home) multi-source DG performance. In terms of single-source DG performance, the proposed approach seems to offer improvements on both PACS and DomainNet (under the experimental settings considered; see weaknesses for more details.). In addition to competitive quantitative results, the ablations in section 4.4.1 demonstrate the utility of different components in the proposed approach.

- The simplicity, intuitiveness and consistent improvements offered by the proposed approach are likely going to make the submission useful for future work to build on top of (provided some of the experimental comparisons are exhaustive; see weaknesses). Overall, I believe this, combined with the lack of reliance on domain labels generally makes the approach valuable. 

I will now highlight major weaknesses associated with the draft below. These are centered mostly around some clarity issues and lack of some experimental comparisons.

 - L178-179 states that the margin m (an important component in the proposed approach) in the objective used to update the novel domain augmentor is “calculated by the L2 distance between the means of the sampled source domains”. It’s unclear to me what this means -- is this the average L2 distance b/w the means across all possible pairs of source domains in the mini-batch? If not, could the authors comment on how exactly this is computed for the multi-source case? Similarly, in the single source generalization setting, m is treated explicitly as a hyper-parameter. However, it’s unclear how the best m is chosen for the single source experiments. Is the best m chosen based on target domain performance? If not, could the authors comment on how this is chosen? 

- Right now, there’s only one line in the paper (and supplementary) detailing how the novel-domain augmentor is realized (as an image translation FCN network) and is hard to locate. The paper would benefit from discussing the architecture and pre-training details (if any) of G earlier section 3. Did the authors experiment with other generators as choices for the augmentor design? Including a discussion about the choice of augmentor architecture (image translation networks vs other plausible generators) would further benefit the paper. 

- While (a) Table 5 clearly highlights the progressive improvements offered by the novel domain augmentor + the teacher-student (EMA) representation learning scheme over RandAugment and JigSaw and (b) Tables 3 & 4 highlight improvements over other data-generation DG approaches, the paper would benefit from discussing recent work [A] (evaluated solely on Single Domain Generalization, but still applicable here) in related work -- particularly since the broad components of the proposed approach in [A] and the current submission seem similar, including plausible O.O.D. instance generation & ensuring similarity across representations from augmented and non-augmented samples. This will further help highlighting the novelty and benefits of the proposed approach over [A].

- Since the proposed approach does not rely on domain labels, single domain generalization benchmarks are somewhat best suited to assess the utility of the proposed approach against other existing single DG approaches. While the authors conduct single DG experiments on PACS and DomainNet (more exhaustive on PACS, as highlighted in the supplementary) and compare with M-ADA, the paper would benefit from including comparisons with [A,B] (which outperform M-ADA) either on PACS / DomainNet or on the benchmarks used [A,B] -- CIFAR-10C or DIGITS or the semantic segmentation benchmarks.

- [Minor Points] -- It’s unclear what L63-64 is trying to convey in the introduction, revising the same would improve the paper. It’s unclear what L228-229 is trying to convey. Not a major point, but I think the opening line of the abstract should be repurposed to replace “task” with “models trained for a task”, it seemed confusing otherwise. The current draft would benefit from a little proofreading to address such minor issues.


[A] - Progressive Domain Expansion Network for Single Domain Generalization

[B] - Uncertainty-guided Model Generalization to Unseen Domains

**Post-Rebuttal Update**
My primary concerns about the paper (centered mostly around clarity and experimental comparisons) have been satisfactorily addressed. I also appreciate the detailed responses to the concerns raised by other reviewers. Given this, I am increasing my rating of the paper to 7.
limitations_and_societal_impact: The strengths and weaknesses highlighted in the review form the basis of my rating. I think addressing the points highlighted under weaknesses, especially the clarification and experimental concerns will help improve the paper. I would encourage the authors to do the same.

I don't see any potential negative social impacts of the work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes Adversarial Teacher-Student Representation Learning to learn domain invariant features for domain generalization. The method learns teacher-student co-training and novel-domain data augmentation in an adversarial manner. 
main_review: The paper is well-written and easy to understand.

Authors proposes the data augmentation to generate synthetic data for domain generalization problem. Such data augmentation is used in domain adaptation and few shot classification often and it is interesting that authors incorporate this idea with domain generations. 

The formulation of minimizing the data discrepancy when training the teacher-student network and maximizing the data discrepancy when updating the data augmenter. Authors evaluate their methods on multiple domain adaptation dataset collections with different network backbones. For the data augmentation network G, authors proposes to augment the input image which requires a large augmentation network. But augmenting extracted features instead of the input image is widely used in data hallucination [A], [B]. It is unclear of the reason why authors choose to augment the image.

The main experiments shows the proposed methods achieve better average acc in domain generalization. Ablation studies show the effectiveness of adversarial augmenter and domain generalized teacher. Visualization provides some generated images. It is shown that the data augmenter generates re-styled images based on the input. The re-styling is visually similar to the re-coloring instead of content preserved style transfer [C]. I am not sure with this kind of data augmentation whether the augmenter succeeds in generating out-of-distribution data and benefits the generalization.

Some questions:
1. Why do you choose to augment the image rather than features?
2. Instead of re-coloring image, is there any other augmentation generated by the augmentor?
3. Is there any noise as input the augmentor in order to generate multiple outputs with the same image?

[A] Wang, Yu-Xiong, et al. "Low-shot learning from imaginary data." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[B] Pahde, Frederik, et al. "Discriminative hallucination for multi-modal few-shot learning." 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 2018.
 
[C] Huang, Xun, et al. "Multimodal unsupervised image-to-image translation." Proceedings of the European conference on computer vision (ECCV). 2018.

Update: Thanks for the authors' response. It addresses my concerns described above and explains very clearly the difference between FSL hallucinations and DG's. I think it is good to include such discussion in the final version, either the main paper or supplementary. I updated the rating. 
limitations_and_societal_impact: Yes, they mention it in section 2.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The article proposes an approach for domain generalization (DG) based on adversarial augmentation and knowledge distillation. In particular, the model contains three components: a teacher, a student, and an augmenter. The student receives input processed by the augmenter while the teacher the original data, and a discrepancy loss measures the difference between student and teacher features for a given sample. Training is held out in two cyclic stages: in the first, the augmenter is fixed and the student parameters are learned by minimizing the semantic and discrepancy loss, with the teacher being an exponential moving average of the student. In the second stage, student and teacher are fixed and the augmenter is updated by minimizing the semantic classification loss and maximizing the discrepancy one. The teacher is used at test time. Experiments on a variety of benchmarks show the efficacy of the proposed approach.
main_review: Overall:
The paper proposes a relatively simple yet effective approach for domain generalization, merging the strength of Mean-Teacher [20] with adversarial data augmentation [8], showing non-marginal improvements over the state of the art in multiple datasets and in both single and multi-source domain generalization. Since both distillation and augmentation strategies are new in the context of domain generalization and I did not spot any major weakness I lean toward the acceptance of the manuscript. Below I detail my comments.

Strengths:
+ Exploiting the consistency between the teacher and the student predictions to train the data augmentation network is interesting since 1) standard DG approaches for domain generalization use a domain classifier (e.g. [7,8]); 2) does not require any domain information; 3) exploits the same network used to predict the semantic of the input, being computationally less demanding.

+ Not many multi-source DG models show to be effective in the single-source DG task (one example is [28]). The proposed approach outperforms existing generative and non-generative approaches in both settings, sometimes by a margin (e.g. +~2% on the challenging single-source DG in DomainNet, Table 2 and +1.2% on multi-source DG in Office-Home, Table 4).

+ The ablation study (Table 5) thoroughly analyzes the impact of each component and design choice.

Weaknesses:

1. The article does not fully specify some of the methodological components, even if it is easy to understand their role. In particular, the classification loss is indicated in Figure 1 but briefly mentioned in the text (line 183).

2. The organization of the experiments can be improved. For instance, multiple tables are used for the comparison with the state of the art, containing redundant results, with the main difference being on the competitors (e.g. Table 1-3 and Table 2-4). It would be good to keep one table for each tested dataset while specifying which methods are generative and which are not. The saved space can be used to either integrate other analyses from the supplementary (e.g. B3, C) or add more qualitative augmentation results. 
limitations_and_societal_impact: No limitations are discussed as well as possible negative social impacts. While I deem the latter to not be present, it would have been important to discuss the limits of the work in terms of, e.g. variety of augmentations produced (Figure 3 shows mostly color-based transformations) and future improvements of the single components.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new data augmentation method for domain generalization. The main idea is to train a neural network-based generator to generate novel images that can be correctly classified by a student model and in the meantime, make the feature discrepancy larger between the student model and the EMA teacher model. The student model, which is fed with the generated images, is trained to minimize both the classification error and the feature discrepancy with its teacher. Only the EMA teacher is deployed at test time. This method is evaluated on both multi- and single-source domain generalization settings, outperforming previous SOTAs by a clear margin.
main_review: **Strength**

- Though learning-based data augmentation has been studied in domain generalization, the idea of using the teacher-student paradigm proposed in this work is novel and could add value to this line of research.

- The evaluation is comprehensive and sufficient, which covers both the multi- and single-source settings as well as different network architectures. The results are encouraging (clearly better than previous data augmentation methods).

**Weakness**

- Section 2 lacks discussions on the differences between this work and existing data augmentation methods. In addition, I do not see a strong connection between this work and SSL methods—except those based on multi-view learning (contrastive learning).

- It's unclear why the single-layer classifier is trained and then fixed during the entire learning process. Why not train the classifier along with the student model?

- The loss formulation in Eq.4 is a bit strange. Once the feature discrepancy is smaller than the margin, no gradients will be backpropagated due to the max(x, 0) function. Why not simply maximize Eq.1 for maximizing the feature discrepancy?

- Compared with confusing a domain classifier or maximizing a distribution distance for synthesizing novel-domain images, the design of maximizing the feature discrepancy between a model and its EMA is not that intuitive. More insights should be provided to explain how this design helps the generator produce images that deviate from the source distributions.

- Line217 says that the generator's architecture is similar to that used in CycleGAN. I'd like to confirm if this generator is trained with only the classification loss and the feature discrepancy loss? Did the authors use other losses like the L1 loss to train the generator?

- In Fig.3, why there are two generated images given the same input? Isn't the generator a deterministic model?

- Line218: Why pretrain the EMA teacher model while giving a random initialization to the student model?

- Line290: What is the difference between Student with EMA and the proposed model (G + F_T)?

---
Post-rebuttal update:

I have gone through the authors' responses, which have addressed my concerns. I have read other reviewers' comments as well.

The proposed data augmentation approach is novel in DG and well justifed with extensive experiments and ablation studies. My concerns raised during the initial review were mainly related to the presentation and clarity, which have been cleared in the rebuttal. I think this paper is worth sharing to the DG community so I change my rating to clear accept.
limitations_and_societal_impact: The authors have properly discussed the limitations of this work in Sec.2, but do not talk about potential negative societal impact. Since this work is a data augmentation method based on learning a generator network from data, the learning process might be biased toward specific datasets used.
review_title: null

Meta-review: This paper was recommended for acceptance by all reviewers. The reviewers appreciated the fact that the paper was well-written, had comprehensive experiments, and presented a simple yet intuitive method. Though the reviewers originally had a few questions, the authors' response sufficiently answered those questions and convinced the reviewers of the merit of this work. The paper addresses the problem of domain generalization and the key aspect the reviewers appreciated the most was the relaxation of the assumption of domain labels and the effective performance shown from the method.
Decision: Accept (Spotlight)