Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a novel approach for domain generalization which relies on an early branching strategy to learn domain invariant features. They show that using HSIC as a domain invariant loss is most effective and also propose a new data augmentation strategy to simulate shifted domains. This work shows that this combination of design decision leads to state-of-the-art performance for domain generalization. While results clearly demonstrating the superiority of early branching for learning domain invariant representation would be quite consequential, the result presented in this work leave room for multiple interpretation as their is significant overlap in performance between branching at over blocks 0-3.
strength_and_weaknesses: This paper presents a new approach for domain generalization through well reasoned design decisions and shows that the method outperforms the strong baseline of ERM and has the highest average accuracy amongst related methods. 

The novel data augmentation strategy seems well reasoned and has strong potential, however it's unclear whether as a standalone it represents a useful contribution. Would be very beneficial to see alternate augmentation strategies combined with losses and branching strategies proposed in this work.  

The results of the ERM+HSIC represents a weakness of this work, it is the core formulation of proposed approach yet shows marginal benefits over the baseline ERM.

A weakness with this work is their seems to be a conflation of hyperparameter search with the domain bed evaluation strategy, some parameters such as branching depth, alpha, and beta are chosen "empirically" in a manner that seems to invalidate the spirit of choosing hyperparameters over random trials in the validation pass over DomainBed. 



clarity,_quality,_novelty_and_reproducibility: The paper is well-written and the method is well designed. 
The quality of ablations could be substantially improved. The paper does demonstrate that each component of the proposed approach positively contributes to the results, however it doesn't effectively show that these components are superior to other reasonable alternatives. 

Based on the paper I believe the work would be straightforward to implement and reproduce the results.
summary_of_the_review: This is an interesting work with components that might be useful for future work in domain generalization, a very important problem. 

However, presenting the results as combination of components makes its hard to distinguish if any individual component represents a meaningful and transferrable contribution. As the main goal is to improve the ability of machine learning model to generalize to unseen domains and not increase placement on leaderboards such as DomainBed its hard to determine the utility of this works contributions. 

More detailed exploration of either Early Branching or RDS could greatly improve this work even if it doesn't yield a new state-of-the-art number.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a domain dual branch network for domain generalization. The first contribution is to empirically verify which layers should be shared between domain and target classification. Then, authors propose a new augmentation strategy for domain augmentation. Experiments using ResNet-18 demonstrates its effectiveness.
strength_and_weaknesses: ### Strength

1. Identifying the sharing and separate layers for dual-branch networks is interesting idea.
2. The paper is well-written.
3. The experiments are sufficient, with good performance.

### Weakness

1. The approach is not novel. Empirically finding the sharing layers can not be regarded as a technical contribution. For the second contribution (new augmentation), it is novel, but still very naive.
2. In page 4, authors spent too many contents explaining the initial results of identifying sharing layers, which I think is useless. It is just experimental observation. Plus, it might not be the case for other backbones like ResNet-50 and Vision transformer.
3. With regards to backbones, authors used DomainNet, which is good, but they only experiment on ResNet-18, which is rather outdated these days since ResNet-50 is the standard and more popular in DomainBed.
4. Identifying which layers to share is ad hoc. You should do everything again in face of a new backbone. This is the limitation of this work and cannot be called a contribution.
5. It will be benfeicial to see the actual running time of this approach compared to others.
clarity,_quality,_novelty_and_reproducibility: Most of the parts are clear and well-written. Novelty is limited. Reproducibility is fine although there's no code, but one can easily reproduce it.
summary_of_the_review: This approach is on the right track: identifying the sharing and non-sharable layers between dual-branch networks. But the method is rather ad hoc and I don't think augmentation part is related to this paper. Plus, experimental results are not sufficient since only resnet18 is used.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tries to solve Domain Generalization (DG) classification problem following the simple idea of decoupling the domain-invariant feature and domain-specific feature by applying an early branching strategy, a new independence measurement HSIC, and a new feature-level data augmentation method generating samples of new domain. Comprehensive experiments on several DG classification benchmarks demonstrate the effectiveness of the proposed method.  
strength_and_weaknesses: Pros:
1.	The logic of this paper is rigorous, and the structure is clear.
2.	The ablation study shows that the HSIC measurement is a better choice compared to the orthogonal constraint and the correlation constraint.
3.	The authors proposed a constrain to obtain inhomogeneous styles to synthesize new domain information.
4.	Experiments on benchmarks show the effectiveness of the proposed method.

Cons:
1.	The authors try to demonstrate early branch architecture has a relatively better performance. However, in the comparing process of different branching location in the network, the number of parameters of shared base feature extractor and two branches are always varies. Thus, such a comparison cannot eliminate the number of parameters’ influence on the performance.
2.	About augments ablation study in Table 3. When the photo or carton is the target domain, the performance of RDS with proposed sampling scheme has no noticeable improvement compared to naive sampling without Eq. (7) , which should be discussed.
3.	Some new works are not discussed in the paper, e.g., Style Neophile: Constantly Seeking Novel Styles for Domain Generalization (CVPR 2022).
4.	The best results in the experiment table should be in bold type to let others find the results easily. Also, the second-best results should be underlined. 
clarity,_quality,_novelty_and_reproducibility: This paper is well-structured and explains the method clearly. The method is clear and looks like easy to reproduce.
summary_of_the_review: The paper follows a simple idea has limited novelty. But the proposed methods make this simple idea more effective. Comprehensive experiments show the effectiveness of early branching and HISC measurement. This method brings new ideas or approaches to handle how to generate novel domain samples and combined it with previous method to form a feature level data augmentation method.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents an interesting study on disentangled representation learning for domain generalization. Technically, the main idea is to build a two-branch network, one for target classification while the other for domain classification. The paper uses carefully designed experiments to demonstrate that separating the two branches at an early stage of a neural network is more effective in learning disentangled representations.

To motivate the method design, the paper starts from a basic learning framework, which combines a classification loss with a correlation minimization loss (the latter is used for learning disentangled representations). Then, the paper motivates the use of domain augmentation (i.e., MixStyle) to facilitate the disentanglement, followed by the introduction of a modified domain augmentation strategy.

The main experiments are conducted on the DomainBed benchmark where the proposed method outperforms most baselines.
strength_and_weaknesses: **Strengths**

First of all, the paper is well-written and well-organized, and both pros and cons of the proposed method are properly discussed in the paper. The idea of learning disentangled representations isn't new in domain generalization. But the study is novel and provides useful tips to the field. The entire framework—including the two-branch network, the correlation minimization loss, and the domain augmentation strategy—is well-motivated. The results clearly show that such a simple method works well in improving model generalization. The two measures, Top-5 and Score used in Table 5, deserve praise and could be of interest to the DomainBed community (where how to properly quantify progress remains an open question).

**Weaknesses**

Overall, there is no major issue that could lead to rejection of the paper. Only a few minor comments are given below, which the authors can use to further improve the paper.

1. The proposed domain augmentation strategy in 3.3 is quite similar to the following work, "Uncertainty Modeling for Out-of-Distribution Generalization" published in ICLR'22, which has been cited in the paper. The current discussions in the end of 3.3 aren't very convincing as both methods are based on random sampling. Could the authors point out the major differences, perhaps using math equations?

2. Have the authors tried other correlation minimization losses besides HSIC? Having more discussions about this could greatly help practitioners choose which loss to use.

3. The Augmentation part in Sec. 2 misses those work using generative modeling to achieve data augmentation, e.g., [1, 2].

4. The 2D t-SNE visualization isn't fully convincing because the spreading of the features could well be manipulated by using different parameters in the t-SNE method. Would it be possible to provide the following visualization: reduce the feature dimension to 2 in the last layer and plot the features directly on a 2D plane?

[1] Zhou, K., Yang, Y., Hospedales, T., & Xiang, T. (2020, August). Learning to generate novel domains for domain generalization. In European conference on computer vision (pp. 561-578). Springer, Cham.

[2] Carlucci, F. M., Russo, P., Tommasi, T., & Caputo, B. (2019, October). Hallucinating agnostic images to generalize across domains. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) (pp. 3227-3234). IEEE.
clarity,_quality,_novelty_and_reproducibility: **Clarity**: The paper is well-written and easy to follow.

**Quality**: The results provide some new insights regarding how to design a better two-branch model for learning disentangled representations.

**Novelty**: The design of the two-branch neural network is well-motivated with numerical results as back-up. The combination of the disentangled feature learning loss and the domain augmentation method is novel.

**Reproducibility**: The implementation detail has been given in the paper.
summary_of_the_review: I recommend to accept the paper because the method is simple, well-motivated and well-evaluated. The findings could be useful to future work on learning disentangled representations for domain generalization.

== Post-rebuttal update ==

I have read the rebuttal as well as other reviewers' comments. The authors have done a good job in addressing the questions I raised in the first-round review. My view on the novelty and significance of the paper remains the same: the paper should be accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: We had a video call with reviewers to discuss this paper and reach a conclusion that this paper still needs further improvement. First, the observation that we need an early branching strategy to learn domain invariant features is heuristic. It would be great if authors can provide more principle guidelines to choose which layer to branch out the network and how this is sensitive to different network architectures with different depth. Second, the proposed augmentation technique is not well motivated by the major observation introduced in the beginning. Instead, they are more like two independent components which can empirically improve the performance.
justification_for_why_not_higher_score: See above.
Decision: Reject