Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a domain generalization method. The key idea is to remove domain-specific features by first training domain-specific classifiers for all domains then training encoder-decoder network that transforms an input image to a domain-invariant version based on them. Specifically, the encoder-decoder network is trained together with another domain-invariant classifier so that the domain-specific classifiers cannot discriminate the classes of transformed images but the domain-invariant classifier can classify them. Experiments show that the model improves the generalization performance from the baselines on PACS, VLCS, and Office-Home with several different backbone architectures.
strengths_and_weaknesses: 1. Strengths

- The idea to learn domain-invariant features has been studied in several existing work. Among them, the paper looks similar to the Epi-FCR [28], which also uses domain-specific classifiers to train a domain-invariant feature extractor. This paper explicitly designs a training scheme to remove domain-specific features and the effect looks pretty clear.
- The proposed method consistently improves the baseline on PACS, VLCS, and Office-Home with different backbone architectures.
- The paper is clearly written and easy to follow.

2. Weaknesses
- I think maximizing the classifier uncertainty of a domain-specific classifier does not necessarily guarantee that the domain-specific features are removed. For example, assume an input image has two channels and the trained domain-specific classifier F_1 uses only the first channel. Since the classifier is fixed, the encoder-decoder network can cheat to include domain-specific features in the second channel, so that it can perform well on the classifier F, which was originally intended to be domain-invariant. At the same time, encoder-decoder network can be trained to output the first channel constant so that the classifier F_1 cannot discriminate classes properly.
- Minor weakness is that the method needs to pass the input image through the encoder-decoder network and it makes an overhead in inference time.

questions: - It would be interesting to visualize output of the encoder-decoder network. It will provide more insights how the method behaves.
- How does the encoder-decoder architecture affect the performance?
- Which data augmentation was used for training? Was the same augmentation used for all the comparing methods?
limitations: The authors did not address the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a novel framework, namely learning and removing domain-specific features for generalization (LRDG), allowing the trained model can extract only domain-invariant features in order to improve the out-of-domain generalization performance. In particular, in LRDG, for each source domain, the training process of the domain-specific classifier is based on the classification loss and the uncertainty loss on other source domains. The experimental results show that the proposed method can produce better classification accuracy compared to several existing methods.
strengths_and_weaknesses: Originality: The proposed framework for learning domain-specific features based on an autoencoder and uncertainty loss in the paper is novel in domain generalization. 

Quality: The technical contributions of the paper are relatively insignificant due to the lack of a mathematical definition for domain-specific features and a theoretical guarantee for the capability to learn the real domain-specificity of the proposed LRDG. Although I appreciate the theoretical result about the generalization bound in Theorem 2, it does not support the main claim of the paper.

Clarity: The paper is quite well-written and easy to follow. 

Significance: My major concern is about the real benefit of the proposed method. In particular, compared to the direct domain-invariant learning approaches (see e.g., [1,2]) the LRDG framework seems to be more complicated and more computationally expensive since requires the training of an autoencoder.  Moreover, the main idea of LRDG is to learn and eliminate the domain-specific features, while some recently published papers (e.g., [3]) show that effectively combining domain-specific and domain-invariant features are also useful for domain generalization. 

[1] @inproceedings{li2018deep, title={Deep domain generalization via conditional invariant adversarial networks}, author={Li, Ya and Tian, Xinmei and Gong, Mingming and Liu, Yajing and Liu, Tongliang and Zhang, Kun and Tao, Dacheng}, booktitle={Proceedings of the European Conference on Computer Vision (ECCV)}, pages={624--639}, year={2018} } 

[2] @inproceedings{hu2020domain, title={Domain generalization via multidomain discriminant analysis}, author={Hu, Shoubo and Zhang, Kun and Chen, Zhitang and Chan, Laiwan}, booktitle={Uncertainty in Artificial Intelligence}, pages={292--302}, year={2020}, organization={PMLR} }

[3] @article{bui2021exploiting,
  title={Exploiting domain-specific features to enhance domain generalization},
  author={Bui, Manh-Ha and Tran, Toan and Tran, Anh and Phung, Dinh},
  journal={Advances in Neural Information Processing Systems},
  volume={34},
  pages={21189--21201},
  year={2021}
}
questions: 1. A mathematical definition for domain-specific features?
2. A theoretical guarantee capability to learn the real domain-specificity of the proposed LRDG?
3. A comparison with the method in [3], which indicates the usefulness of domain-specific features in domain generalization?
limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The manuscript describes a new framework for training models that are general for multiple domains (domain generalisation). Their protocol is to (1) train one classifier for each domain, which are made domain-specific by enforcing a random guess for domains that they are not specialised to, (2) train an auto encoder where the decoding process is constrained to generate images that “confuse” all of the domain specific classifiers and at the same time (3) train a new classifier on top of those domain-agnostic images.
strengths_and_weaknesses: The paper is well written and easy to follow. The presented idea for generating domain-invariant images and training domain specific classifiers is intuitive and the entire package seems to bring about improvements on average to performance on the baselines datasets. The authors have made comparisons against state-of-the-art studies and compared across different backbones.

The use the generalisation bound, and of the Proxy A-distance (PAD) is very welcome here for showing that the model truly generalises. However, the application used by the authors employ the features of an AlexNet network for the unmapped domains and the features of their domain-invariant classifier for the mapped domains; this makes it hard to assess whether their mapping was effective in reducing the H-divergence between domains. It would probably be more adequate to use the same features for both to allow for a fairer comparison. I understand this is not a simple choice given the features of an AlexNet will be more discriminative for the photo domain (and this will probably also apply for other pre-trained networks) and the features of the domain-invariant classifier would be less discriminative for unmapped domains since it was trained for mapped versions. 

Even though it is the intuitive conclusion that the framework would remove domain-specific features, it is not guaranteed to do so. Different classifiers could work with different manifolds of the original image space to do their classification and the uncertainty loss could force each classifier to use their own “space” to perform classification, ignoring or not domain-specific features from other domains; this is similar to how SVMs with different kernels could solve the same classification tasks and even use spaces with the same dimensionality. This non-guarantee is important to state since the authors state that they reduce \lambda (lines 174-176) and “prove” that they reduce e in the generalisation risk bound; maybe those statements could be toned down with “probably” or “indicates that … reduces”. 
questions: Why does Figure 2 not include PAD of pairwise domains with Photo and Cartoon as target domains? 
 
Did the authors consider performing a similar method on a latent space instead of directly on the image space? (This question or its answer do not interfere with my rating)

limitations: The limitations of the model are addressed in the Conclusion. The authors discuss the need for training separate classifiers for each source domain and that the model is not able to remove domain-specific features that regard unseen domains. I would add the limitation that some domain-specific knowledge is useful for some domains, as we can see that this model does not outperform others on more complex domains such as Photo and sometimes Art.

EDIT (post-discussion): I will also add this here for completeness. Per the discussion below and with other reviewers, the authors have acknowledged that the removal of domain-specific features is conditional on these features being learned by the domain-specific classifiers via the optimisation in Eq. (8). This is expected to work given the design of Eq. (8) but it is not *guaranteed*, and therefore this assumption configures an expected limitation of the work that future readers and users should keep in mind.
review_title: null

Meta-review: metareview: After the author-reviewer discussion, Reviewer 8K1L shows strong support for the paper, and Reviewer MBUN finds most concerns addressed and upgrades the score to Weak accept. Reviewer T9VV has some remaining concerns, but does agree the proposed method seems working empirically. After careful consideration, AC recommends accepting the paper.
confidence: Certain
Decision: Accept