Exploiting Domain-Specific Features to Enhance Domain Generalization

Abstract: Domain Generalization (DG) aims to train a model, from multiple observed source domains, in order to perform well on unseen target domains. To obtain the generalization capability, prior DG approaches have focused on extracting domain-invariant information across sources to generalize on target domains, while useful domain-specific information which strongly correlates with labels in individual domains and the generalization to target domains is usually ignored. In this paper, we propose meta-Domain Specific-Domain Invariant (mDSDI) - a novel theoretically sound framework that extends beyond the invariance view to further capture the usefulness of domain-specific information. Our key insight is to disentangle features in the latent space while jointly learning both domain-invariant and domain-specific features in a unified framework. The domain-specific representation is optimized through the meta-learning framework to adapt from source domains, targeting a robust generalization on unseen domains. We empirically show that mDSDI provides competitive results with state-of-the-art techniques in DG. A further ablation study with our generated dataset, Background-Colored-MNIST, confirms the hypothesis that domain-specific is essential, leading to better results when compared with only using domain-invariant.


Reviews and author responses:

 Dear Reviewer,

Thank you a lot once again for your further feedback. Regarding your remaining concern, we will definitely take it into account in the revised version of our paper.

Have a great weekend,

Authors.

 Thanks to the authors for their details responses and especially the additional ablation results. My concern remains about the novelty of this paper (shared by Reviewer YJwo). I would not change my original rating in this regard.

 Dear Reviewer,

Thanks a lot for your efforts in reviewing this paper. We have tried our best to address all your concerns and provided clarifications on all confusing concepts.

The discussion period is nearing an end, thus we wonder if you can spend some time going over our newest replies as well as our supplementary materials, to see if they successfully answered your questions or not. This is also to give us a decent amount of time to address any of your remaining/additional concerns.

Best regards,

Authors.

 Dear Reviewer,

Thanks a lot for your efforts in reviewing this paper. We have tried our best to address all your concerns and provided clarifications on all confusing concepts. 
As the ending of the discussion period is approaching, please let us know whether there are any unclear explanations. In addition, if you have any further questions, we will also be very glad to further clarify them.

Sincerely thanks,

Authors


 Thank you very much once again for your great feedback and positive evaluation of our paper. We also gratefully appreciate your additional suggestions; and will definitely take them into account in our next paper revision.

All the best,

Authors.


 Apologies for the delay. Thanks to the authors for providing a detailed response. Also, thanks to the other reviewers for pointing out bits that I had missed in my initial assessment of the paper — especially reviewer d6mf for pointing out issues with the setting where performance can suffer when DS features correlate with a different class label and the subsequent instantiation in the “Background-Color-MNIST” dataset.

My primary concerns revolved around 3 major points — clarification surrounding the meta-training procedure, an additional ablation demonstrating the utility of the disentanglement loss, and an experiment consisting of randomizing the domain label to highlight the utility of domain specific features. The first two concerns were addressed satisfactorily. Regarding the third point of concern, the rotated MNIST results seem to indicate “no-change” in performance when domain labels are randomized and the “rest”->”photo” shift in Office-Home seems to imply a degradation in performance. While there is some evidence that domain-specific features are important, from the reported results, it seems that their utility is subjective to the shift involved. I would encourage the authors to include — (1) the overall impact on performance across all shifts (similar to Table 1) when domain labels are randomized and (2) a discussion paragraph highlighting for which characteristic datasets and shifts, domain label randomization has an impact.

Most concerns associated with the paper (including mine and other reviewers) seem to have been addressed / will be addressed in revised versions of the paper. As such, I would prefer to stick to my original rating of the paper.

 Dear reviewers,

We would like to thank you again for spending your time evaluating our paper and reading the rebuttals.

As the ending of the discussion period is approaching, we look forward to hearing your feedback about whether we have addressed your concerns regarding our rebuttals. We would be happy to discuss if you still have any concerns.

All the best,

Authors.

 Dear Reviewer,

We gratefully appreciate your prompt response to our rebuttal, especially for upgrading the score for our paper. We will definitely take our paper limitations into account and will add a more detailed discussion to clarify that in the revised version.

All the best,

Authors.

This paper proposes a framework to learn both domain-invariant (DI) and domain-specific (DS) features for the purpose of domain generalization. The DI features are learned using an existing adversarial method which seeks a representation from which the domain *cannot* be predicted. The DS features are learned through a meta-learning framework, where the outer loop seeks to do well on a held-out validation-set (in-distribution examples). The main contribution is the unified framework in which to learn both DI and DS features. Experiments are conducted on selected datasets of DomainBed, and the results on these selected datasets are strong.   **Summary**: Novel and interesting approach to exploiting DS features. DS features clearly boost performance on datasets where the source confounders (spurious correlations) correlate with *the same class label* in the target/test domain. E.g. if, in a source domain, grass strongly indicates cow and not camel, then using grass in the target/test domain will improve predictions if and only if grass still indicates cow. If instead grass correlates with camels in your target/test domain, then the DS features will *hurt* performance. This was the entire motivation behind Invariant Risk Minimization (IRM, [1]) -- discard DS features as we cannot trust them in the test domain -- and it seems to be completely ignored throughout this paper. While the direction is interesting and the results impressive, I find it inexcusable to ignore (and perhaps bury -- see below) this important caveat/limitation. The severity of failing to explicitly state assumptions/limitations is compounded by the authors’ claim of a “theoretically sound” framework. Thus, I believe the paper is just below the acceptance threshold in its current form, but am *very* willing to increase this score if the authors: (i) add a thorough discussion of the limitations of using DS features; and (ii) explicitly define their assumptions (i.e. explicitly define the situations in which DS features help and hurt performance in the test domain).

**Strengths**: 
- *Interesting and important direction*: combining DI and DS features.
- *Strong empirical results*: on the selected datasets of DomainBed.
- *Very clear writing*. I particularly like the Venn diagrams in Figure 1, the simplicity of Figure 2, and the insights of Figure 3.


**Weaknesses**: 
- *Limitations are ignored, and perhaps even buried*. As described in the summary above, DS features can hurt performance if they correlate with a different class label in the test domain than they did in the source domain(s). This point is mentioned in lines 24-25, but then ignored throughout the rest of the paper. This issue can be divided into two points:
    - Most importantly, the *Color-MNIST* dataset of [1] is not only left out of DomainBed evaluations without explanation (clearly it is left out as DS features will hurt test performance), but the authors introduce their own dataset with the same name, and design it such that the DS features in the test domain *do* help test performance. Specifically, the test domain contains *seen* background colours that are *unique to the class label*, e.g. a particular kind of green always indicates ‘5’ in training domains. This overloading of the name Color-MNIST only serves to conflate two very different datasets, and further buries the limitation of using DS features in the test domain. I am left wondering why:
        - The authors didn’t call it e.g. Background-Color-MNIST to distinguish it from the already-existing Color-MNIST.
        - The authors did not discuss this overloading/conflation -- i.e. why they left out the original Color-MNIST [1] and created their own dataset with the same name.
    - Secondly, there are several points throughout the paper where the authors make grand claims about domain generalization without acknowledging the aforementioned caveat (or, equivalently, the required assumptions for these claims to hold true). For example on lines:
        - *93*: "will limit generalization ability": *only if DS features from the source domains correlate with the same class label in the target domain*.
        - *154*: “better prediction performance": *in domain S^1*.
        - *160*: “inferior prediction performance": *in the source domains*.
        - *163*: “better generalization performance”: under what assumptions? See first point. Clearly this doesn’t hold for e.g. the ColorMNIST dataset of [1].
        - *166-169*: the authors *do* acknowledge that DS features may not be relevant to the target domain. However, they speak about redundancy (present or not) rather than the true limitation of DS features hurting performance by indicating the wrong class.


**Additional feedback/questions** (in order of importance): 
- Adding baselines of ERM (on the pooled source-datasets) and IRM to *your* Color-MNIST dataset would really help put the numbers into context. ERM might do quite well since, in the test domain, the background colours are seen and unique to the class label.
- What hyperparameters were fitted using the validation set for all methods (yours included)? You say you selected the best model using a validation set, but your Table 3 of Appendix B.3 does not contain a final column showing the random search distribution for hyperparameter sweeps (as in Appendix D.2 of [2]).
- Why was the Terra Incognita dataset of DomainBed not included?
- How long did it take to train your model? 
- Is the calculation of the Covariance matrix expensive for large feature dimensionality? Does it require 12 CPUs?
- Figure 3 -- best view in color *on a computer*! Nice figure nonetheless.
- Do you use a *shared* feature-extractor for learning DI features? Is this optimal? See e.g. [3]

[1] Arjovsky, M., Bottou, L., Gulrajani, I., & Lopez-Paz, D. (2019). Invariant risk minimization. arXiv preprint arXiv:1907.02893.

[2] Gulrajani, I., & Lopez-Paz, D. (2020). In Search of Lost Domain Generalization. In *International Conference on Learning Representations*.

[3] Tzeng, E., Hoffman, J., Saenko, K., & Darrell, T. (2017). Adversarial discriminative domain adaptation. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition* (pp. 7167-7176).


***Update after response***: I have raised by score by 1 – see reply to authors' comment.

 No, the authors have not adequately addressed the limitations of their work (see point under "weaknesses" above). Authors selected "N/A" for limitations in the checklist (1b).

 Dear authors,

Thank you for your detailed response and additional results. I have raised my score with the understanding that an explicit discussion paragraph on limitations will be added, which discusses when DS features might help and hurt. I hope this goes behind the speculation of "DS features are unlikely to hurt". When discussing the limitations, it might also be worth adding: (1) the 30gb memory cost of storing of 2048×2048 covariance matrix; and (2) the fact that your method uses 2xfeature extractors (ResNet50s), so perhaps the comparison to methods that only use 1 (ERM, IRM, etc.) is not entirely fair.

 Thank you for your detailed and constructive reviews. We truly appreciate that. We would clarify your concerns below.

### 1. Original Colored-MNIST dataset is left out while creating a different one with the same name.
We apologize for creating a different dataset while keeping a similar name. Thanks for your suggestion; we will rename them to “Background Color MNIST”. As for the original Colored-MNIST, we have done experiments on it and we found performance on this dataset varies a lot between model selection techniques. As mentioned in our paper, we only use the “training-domain validation set” technique for model selection. We provide its actual results here: 

For “training-domain validation set”: 
- IRM [1]: 72.5 ± 0.1 (10% flip), 73.3 ± 0.5 (20% flip), 10.2 ± 0.3 (90% flip)
- mDSDI: 73.4 ± 0.2 (10% flip), 73.1 ± 0.3 (20% flip), 10.1 ± 0.0 (90% flip) 

For “test-domain validation set”: 
- IRM [1]: 72.7 ± 0.1 (10% flip), 72.8 ± 0.3 (20% flip), 65.2 ± 0.8 (90% flip)
- mDSDI: 73.3 ± 0.4 (10% flip), 73.2 ± 0.3 (20% flip), 31.1 ± 0.3 (90% flip)

We agree that our model still concentrates on the color features and has worse performance than IRM [1] in one extreme scenario with the “test-domain validation set” model selection. The color will hurt model performance in this dataset because it will be flipped randomly with a higher probability in the “90% flip color” domain. Therefore, the model should not rely on color features and only focus on digits.

However, you can see that in order to achieve that ability, IRM [1] still has to rely on the target domain during training; in other words, it is simply told that color is not invariant in the target domain anymore and lets ignore them. Otherwise, it will treat color as a domain-invariant feature and also perform badly in the “90% flip color” domain (see results in the “training-domain validation set” model selection above). However, the “test-domain validation set” technique is unrealistic in DG; that is why we decided to ignore them.

### 2. When domain-specific features will *hurt/help* classification performance in the test domain.
Thanks for your question, and we will add a discussion to our paper. The domain-specific features may *hurt* performance if they correlate with a different class label in the unseen domain than they did in the source domains (i.e., $I(Z_{I^*};Y)+\epsilon_1 \neq I(X^T;Y^T)$). Following your mentioned example about cow and camel, if in an unseen domain, a camel stands in a field rather than desert, it will hurt our model performance. Another observation is in our ablation study that if the yellow background of number “2” in the source domain becomes the background of number “1” in the target domain, our model’s classification accuracy on the class number “1” will drop by 40% in the target domain.

However, as discussed in the paper (lines 24-26), we believe domain-specific features likely will *help* classification performance in the real world (i.e., $I(Z_{I^*};Y)+\epsilon_1=I(X^T;Y^T)$). The correlation between different class labels in the unseen than they did in the source domain is unlikely to appear. For example, camels are more likely to appear in the desert than in a field, and cows are more likely to appear in the field than in the desert. Moreover, in Sec. (3.2) in the main paper and Appendix B.4, we mentioned several observations in the real world dataset in which domain-specific features of source domains strongly correlate with a class label in both source and target domains.
 
### 3. Adding baselines of ERM and IRM to your Color-MNIST dataset. 
As requested, we provide our additional comparison after reimplementing ERM and IRM on our Color-MNIST dataset. The result of ERM is "76.2 ± 3.6" while that of IRM is "66.7 ± 3.2". These results are much lower than our mDSDI and quite similar with “DI”, “DI-meta”, “DS”, and “DS-meta” in Table 2. The reason is that, for both ERM and IRM, they do not take both domain-invariant and domain-specific into account, so they only use one Resnet-50 feature extractor. Meanwhile, mDSDI uses two different feature extractors, making the model more complex. That is why we can see a significant discrepancy between them and our model. Of course, besides the model complexity reason, other components in our loss (see Sec. (2.3) and Sec. (3.4)) also boost the performance a lot. For instance, “DSDI-Without Meta” also uses two feature extractors. However, without meta-training on domain-specific, it's also lower than mDSDI around 9% in Table 2.

### 4. What hyperparameters were fitted using the validation set for all methods, why table 3 of Appendix B.3 doesn't show the hyperparameter searching distribution?
We used the exact hyperparameters mentioned in the final column in Table 3 of Appendix B.3. We want to emphasize that we do model selection, not hyperparameter searching, in DomainBed:
- For model selection: We said: “we selected the best model using a validation set” because compared with DomainBed, they have three different methods for model selection, including: “Training-domain validation set” (what we used), Leave-one-domain-out cross-validation, and Test-domain validation set (oracle) (as in Sec. (3.1) of DomainBed). This model selection is practical in real-life situations.
- For hyperparameter searching: We do not use hyperparameter searching because we believe such exclusive model tuning is impractical in handling various real-life tasks, leads to unfair comparisons, and may cause overfitting to the target test domain. Instead, we use the standard hyper-parameter configurations used in the previous works. Therefore, in Table 3 of Appendix B.3, we do not have a final column showing the random search distribution for hyperparameter sweeps.

### 5. Why was the Terra Incognita dataset of DomainBed not included?
The download link for Terra Incognita dataset was broken at the paper submission time. Therefore, we could not report our results on this set in the submitted manuscript. The link has just been fixed on 07/21/2021  (https://github.com/facebookresearch/DomainBed/tree/master/domainbed/scripts). We have run experiments on this set, and the results are provided below:

|Algorithm|L100|L38|L43|L46|Avg|
|-|-|-|-|-|-|
|ERM|49.8±4.4|42.1±1.4|56.9±1.8|35.7±3.9|46.1|
|IRM|54.6±1.3|39.8±1.9|56.2±1.8|39.6±0.8|47.6|
|GroupDRO|41.2±0.7|38.6±2.1|56.7±0.9|36.4±2.1|43.2|
|Mixup|**59.6±2.0**|42.2±1.4|55.9±0.8|33.9±1.4|47.9|
|MLDG|54.2±3.0|**44.3±1.1**|55.6±0.3|36.9±2.2|47.7|
|CORAL|51.6±2.4|42.2±1.0|57.0±1.0|39.8±2.9|47.6|
|MMD|41.9±3.0|34.8±1.0|57.0±1.9|35.2±1.8|42.2|
|DANN|51.1±3.5|40.6±0.6|57.4±0.5|37.7±1.8|46.7|
|CDANN|47.0±1.9|41.3±4.8|54.9±1.7|39.8±2.3|45.8|
|MTL|49.3±1.2|39.6±6.3|55.6±1.1|37.8±0.8|45.6|
|SagNet|53.0±2.9|43.0±2.5|**57.9±0.6**|40.4±1.3|**48.6**|
|ARM|49.3±0.7|38.3±2.4|55.8±0.8|38.7±1.3|45.5|
|VREx|48.2±4.3|41.7±1.3|56.8±0.8|38.7±3.1|46.4|
|RSC|50.2±2.2|39.2±1.4|56.3±1.4|**40.8±0.6**|46.6|
|mDSDI|53.2±3.0|43.3±1.0|56.7±0.5|39.2±1.3|48.1|

Our method provides the second-best result on the average accuracy, only lost to SagNet. Note that this dataset lacks domain-specific features to fully exploit the advantages of our algorithm. It contains photographs of wild animals taken by camera traps; camera trap locations are different across domains. Since these cameras are static, different animals still have the same background, or in other words, there is no correlation between background and animal label in this dataset. Hence, it may lack domain-specific features, making our model rely mainly on domain-invariant features of the object animal.

### 6. How long did it take to train your model?
It depends on the dataset, models, hyperparameters, and computer resources. Given our mentioned hyperparameter (batchsize, epochs) and resource used, we spent around 8 hours for all domains in MNIST-related datasets, 6 hours for each VLCS, PACS, Office-Home, and 7 days for DomainNet.

### 7. Is the calculation of the Covariance matrix expensive for large feature dimensionality? 
Yes, the higher the feature dimensionality, the more expensive for calculating the Covariance matrix. Specifically, if the backbone is Resnet50, we have to store a 2048x2048 covariance matrix which causes our model to consume around 30GB. And the running speed is also affected; in PACS, if we decrease feature dimension from 2048 to 512, the inference time will decrease from around 15s to 11s while training time reduces from about 1h30’ to 1h10’. But this will lead to about a 10% accuracy drop.
 
Given this weakness, we tried an alternative solution with adversarial training to minimize the mutual information between $Z_I$ and $Z_S$. The idea was that we could derive $\min I(Z_I, Z_S) = \min KL(P(Z_I, Z_S), P(Z_I). P(Z_S))$, and if we shuffled $Z_S$ w.r.t. to the index of samples in each mini-batch, we also could disentangle these features without being affected by feature dimension. However, the results had a higher variance in terms of accuracy, so we decided to stick with the Covariance matrix technique.

### 8. Does it require 12 CPUs?
No, it doesn't. We wrote 12 CPUs because it is truly the amount we used to run this experiment.

### 9. Do you use a shared feature-extractor for learning domain-invariant features? Is this optimal? 
Yes, we use a shared feature-extractor for learning domain-invariant features between meta-source and meta-test domain (compared to 2 different extractors in [3]). However, it is worth noting that we use two different feature-extractors for learning domain-invariant and domain-specific features. 

Yes, our domain-invariant feature-extractor is optimal because we observe that the classification accuracy of domain-invariant with respect to $L_{Z_I}$ loss decreases after a few iterations, then remains stable around 30% during the training time. (see lines 676-678 in Appendix B.5).

Thank you for your willingness to increase our score. Please let us know if you would like us to do anything else.

 Thanks for your feedback. We address the main concerns below.

### 1. The novelty of paper: The domain-specific is not new in Domain Generalization, proposed meta-training is not clearly indicated.
We agree with you that utilizing domain-specific features also has been studied in the work of [1] and [2]. However, [1] introduces multiple domain-specific networks for each domain, then uses structured low-rank constraints to align them with domain-invariant. While it encourages a better transfer of knowledge, its main problem is the requirement of too many domain-specific networks. Our work is most closely related to the domain-specific Masks for Generalization (DMG) framework [2], which integrates both domain-invariant and specific information. However, one of its key drawbacks is that domain-invariant/domain-specific representations might not be disentangled since the learning and inference procedures are performed implicitly, i.e., without any theoretical guarantee, through a mask generalization process. This causes both the domain-invariant features, assumed to be stable across training domains and domain-specific features to be ineffective on the unseen domain.

To handle these shortcomings, in this paper, we propose a novel Domain Generalization approach that can extract domain-specific and then explicitly disentangles the domain-invariant and domain-specific representations in an efficient way without the need to train multiple networks for domain-specific. We develop a rigorous framework to formulate elements of domain-invariant/domain-specific representations and develop necessary theoretical insights for it. Our key insight is to introduce an effective meta-optimization training framework to learn domain-specific representations from multiple training domains. Without access to any data from unseen target domains, the meta-training procedure provides a suitable mechanism to self-learn domain-specific representation.

[1] Ding, Z. and Fu, Y. Deep domain generalization with structured low-rank constraint. IEEE Transactions on Image Processing, 27(1):304–313, 2018. doi: 10.1109/TIP.2017. 2758199.
 
[2] Chattopadhyay, P., Balaji, Y., and Hoffman, J. Learning to balance specificity and invariance for in and out of domain generalization, 2020.
 
### 2. No supplementary material is provided to guarantee theoretical results.
Please recheck our supplementary PDF in the zipped file. We did provide proofs of our Lemma and Theorem.
 
### 3. Ablation studies with one of the 5 benchmark datasets.
We mentioned a similar ablation study with PACS benchmark dataset is in Appendix B.6 (lines 303-304 in the main paper).
 
### 4. Acronyms are hard to follow and reference papers are not in the standard citation format.
Thank you for your constructive feedback. We will correct them in the revised version.

We hope you can reconsider your given score after seeing our supplementary materials. Please let us know if you would like us to do anything else.

 Thanks for your feedback. We address the main concerns below.

### 1. Novelty of paper: the disentanglement of domain-invariant and domain-specific features have been proposed in DG [1].
Thanks for your question. While [1] also takes domain-invariant and domain-specific into account, it is used for Unsupervised Domain Adaptation, a task completely different from Domain Generalization (DG). Please note that it is strictly not allowed to access the target domain during the training process in the DG setting.

In DG, utilizing domain-specific features also has been studied in the work of [5] and [6]. However, [5] introduces multiple domain-specific networks for each domain, then uses structured low-rank constraints to align them with domain-invariant. While it encourages a better transfer of knowledge, its main problem is the requirement of too many domain-specific networks. Our work is most closely related to the domain-specific Masks for Generalization (DMG) framework [6], which integrates both domain-invariant and specific information. However, one of its key drawbacks is that domain-invariant/domain-specific representations might not be disentangled since the learning and inference procedures are performed implicitly, i.e., without any theoretical guarantee, through a mask generalization process. This causes both the domain-invariant features, assumed to be stable across training domains, and domain-specific features to be ineffective on the unseen domain.

To handle these shortcomings, in this paper, we propose a novel DG approach that can extract domain-specific and then explicitly disentangles the domain-invariant and domain-specific representations in an efficient way without the need to train multiple networks for domain-specific. We develop a rigorous framework to formulate elements of domain-invariant/domain-specific representations and develop necessary theoretical insights for it. Our key insight is to introduce an effective meta-optimization training framework to learn domain-specific representations from multiple training domains. Without access to any data from unseen target domains, the meta-training procedure provides a suitable mechanism to self-learn domain-specific representation.

[1] X Peng, Z Huang, X Sun, and K Saenko. Domain Agnostic Learning with Disentangled Representations. In International Conference on Machine Learning, 2019.

[5] Ding, Z. and Fu, Y. Deep domain generalization with structured low-rank constraint. IEEE Transactions on Image Processing, 27(1):304–313, 2018. doi: 10.1109/TIP.2017. 2758199.

[6] Chattopadhyay, P., Balaji, Y., and Hoffman, J. Learning to balance specificity and invariance for in and out of domain generalization, 2020.

### 2. Novelty of paper: adversarial training and meta-learning already be used in [2, 3, 4], which are not cited in the paper.
Thanks for your question. We agree that there are several papers that use adversarial training and meta-learning techniques in DG; and have already mentioned some typical ones in our paper, such as MLDG [11], DANN [30], and CDANN [31]. We agree that three of your mentioned papers are missing in our paper; the reason is that we can not reproduce their experimental results, which was raised in DomainBed [7] too. We will add those mentioned papers in the related work in our paper.

It is important to emphasize that none of those methods has combined these two techniques in Domain Generalization; and our paper is the first work to propose the use these of two techniques effectively by applying adversarial training for domain-invariant and meta-learning for domain-specific. The dominant adversarial learning only focuses on learning invariant representation while ignoring the useful domain-specific, leading to sub-optimal generalization performance. Inspired by this observation, our paper introduces a method that aims to incorporate useful domain-specific and domain-invariant features, targeting a better classification accuracy on the unseen domain and developing necessary theoretical insights. It is worth noting that the meta-learning procedure first employed in our method to learn informative and useful domain-specific features is totally different from the one you mentioned (see footnote 1).

[2] Q. Dou, D. C. Castro, K. Kamnitsas, and B. Glocker. Domain Generalization via model-agnostic learning of semantic features. In Advances in Neural Information Processing Systems, 2019. 

[3] S. Zhao, M. Gong, T. Liu, H. Fu, and D. Tao. Domain Generalization via entropy regularization. In Advances in Neural Information Processing Systems, volume 33, 2020.

[4] Y. Balaji, S. Sankaranarayanan, and R. Chellappa. MetaReg: Towards Domain Generalization using meta-regularization. In Advances in Neural Information Processing Systems, 2018.

[7] Gulrajani, I., & Lopez-Paz, D. (2020). In Search of Lost Domain Generalization. In the International Conference on Learning Representations.

### 3. Little modules about the information theory in the implementation.
Despite not exactly minimizing measures of mutual information as discussed in information theory (which causes a high computational cost in DG), our implementation still has a strong connection with the theory section, following these reasons:
- The classifier within standard cross-entropy minimization in Eq. (2) and  Eq. (5) has been shown to be similar to minimal and sufficient representation with the label (definition (3)) in [8]. This is also a fundamental definition of the information bottleneck method, which is proved in [9].
- The domain classifiers within an adversarial training framework in Eq. (1) can be used as a proxy for the minimal and sufficient representations with domain-invariant in definition (4). Because following definition (4) in our paper, it maximizes mutual information across source domains and minimizes redundancy specific information in a particular domain. Its optimal solution is similar to the adversarial training framework in Eq. (1).
- In the disentanglement loss in Eq. (3), minimizing Covariance between two random variables is equivalent to minimizing mutual information between them. The reason is: we can derive min $I(Z_I, Z_S) = min KL(P(Z_I, Z_S), P(Z_I).P(Z_S))$. Two random variables X, Y are disentangled (independent) if covariance (X, Y) = 0. Hence, if they are feature vectors $Z_I = [Z_{I1}, Z_{I2}, ..., Z_{Im}]$ and $Z_S = [Z_{S1}, Z_{S2}, ..., Z_{Sn}]$, they will be disentangled (independence) if $COV(Z_{Ii}, Z_{Sj}) = 0$ for every (i, j) in the covariance matrix.

[8] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self- supervised learning from a multi-view perspective. In International Conference on Learning  Representations, 2021.

[9] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.

### 4. Why do the domain-invariant and specific features need to be disentangled first and then aggregated for prediction?
As mentioned in the paper, we must disentangle domain-invariant and domain-specific features to optimize in different ways (see Sec. (2.3) and Fig. (2)). Domain-invariant is stable across domains, so we apply adversarial training to learn; while domain-specific belongs to a particular domain, and the feature extractor needs to learn how to transfer useful domain-specific information from source domains to unseen domains (see footnote 1). Therefore, we apply meta-learning to train the domain-specific extractor to gain that ability (see lines 203-211 in the main paper).

### 5. How about jointly learning the features without disentanglement and aggregation?
Thank you for your question. Without disentanglement and aggregation, it is unable to distinguish domain-invariant and domain-specific features. We, therefore, can not apply adversarial training and meta-learning on the same features because it may apply adversarial training for domain-specific, meta-learning for domain-invariant, this will hurt model performance.

### 6. More analysis and supportive experiments are needed.
We analyzed our model and its behavior in different settings (including removing meta-learning, applying meta-learning for domain-invariant, etc.) in Sec. (3.4). Please also see several supportive experiments for our proposed model in Sec. (3.2) and Sec. (3.3) in the main paper and more detailed analyses provided in Appendix B.4, B.5, and B.6.

### 7. The difference between the extracted domain-invariant $Z_I$ and domain-specific features $Z_S$ is not clear. What would happen if the $L_D$ term in Eq. (8) was removed?
As mentioned in “Domain-Invariant and Domain-Specific Extraction” in Sec. (2.3) (lines 185-193), we do adversarial training to optimize $Z_I$ (it is min max term), and apply standard supervised learning to minimize cross-entropy to optimize $Z_S$. Furthermore, $Z_S$ is optimized during the meta-training process while $Z_I$ is not (see lines 203-211 in the main paper). 

And $L_D$ term is used to disentangle these latents $Z_I$ and $Z_S$, if without $L_D$ in Eq. (8), there is no guarantee these latents are disentangled (independent) or not. To prove our argument, we provide our additional comparison after removing disentanglement loss in Eq. (3) (i.e., $L_D$ in Eq. (8)) in the main paper, and the results are provided below:

- Our Colored-MNIST:
|Method|Accuracy|
|-|-|
|DSDI-without $L_D$|81.4±2.6|
|DSDI-Meta DS (mDSDI)|**89.7±0.8**|

- PACS:
|Method|Art|Cartoon|Photo|Sketch|Average|
|-|-|-|-|-|-|
|DSDI-without $L_D$|86.4±0.5|78.9±0.8|98.0±0.3|72.4±2.4|83.9|
|DSDI-Meta DS (mDSDI)|**87.7±0.4**|**80.4±0.7**|**98.1±0.3**|**78.4±1.2**|**86.2**|

These results show that without disentanglement loss, the model always performs worse than mDSDI. It reveals adding disentanglement loss is essential to boost our model performance.

We hope that you can reconsider the review score. Please let us know if you would like us to do anything else.

 Thank you for your detailed reviews and your compliment on our idea with Domain Generalization from an Information Bottleneck perspective. We truly appreciate that. We would clarify your concerns below and make changes in the revision version as you suggest.
### 1. Meta-training sampling process is confusing: Does this mean instances from domain i in a mini batch are split into two groups? Or that k-source domains in a minibatch are split into k-1 meta-train and 1 meta-test domains?.
We clarify that we are following the latter. Specifically, in each mini-batch, k-source domains are split into k-1 meta-train and 1 meta-test domains.
 
### 2. Adding without disentanglement loss in additional ablation study.
As requested, we provide our additional comparison after removing disentanglement loss in Eq. (3) (i.e., $L_D$ in Eq. (8)) in the main paper, and the results are provided below:

- Our Colored-MNIST dataset:
|Method|Accuracy|
|-|-|
|DI|65.7±4.6|
|DI-Meta|63.6±5.1|
|DS|70.7±4.8|
|DS-Meta|75.3±3.4|
|DSDI-without Meta|80.4±1.7|
|DSDI-Meta|82.1±1.4|
|DSDI-Meta DI|79.0±2.3|
|DSDI-without $L_D$|81.4±2.6|
|DSDI-Meta DS (mDSDI)|**89.7±0.8**|

- PACS dataset:
|Method|Art|Cartoon|Photo|Sketch|Average|
|-|-|-|-|-|-|
|DI|84.5±0.6|77.4±0.8|98.0±0.3|74.7±1.2|83.7|
|DI-Meta|84.4±0.5|77.3±0.7|97.5±0.4|75.1±1.4|83.6|
|DS|85.2±0.5|77.9±0.9|98.0±0.3|72.5±1.3|83.4|
|DS-Meta|87.1±0.4|79.2±0.7|98.2±0.3|74.1±1.2|84.7|
|DSDI-without Meta|84.4±0.6|79.2±0.8|98.3±0.3|75.7±1.4|84.4|
|DSDI-Meta|86.5±0.3|78.4±0.7|**98.3±0.3**|77.5±1.2|85.2|
|DSDI-Meta DI|84.5±0.4|77.8±0.8|97.9±0.3|76.6±1.1|84.2|
|DSDI-without $L_D$|86.4±0.5|78.9±0.8|98.0±0.3|72.4±2.4|83.9|
|DSDI-Meta DS (mDSDI)|**87.7±0.4**|**80.4±0.7**|98.1±0.3|**78.4±1.2**|**86.2**|

Overall, these results show that without disentanglement loss, the model always performs worse than mDSDI. It reveals adding disentanglement loss is essential to boost our model performance.
 
### 3. Adding randomized domain labels to validate effectiveness of domain-specific.
As suggested, we quickly implemented this setting on Rotated MNIST (which unlikely contains domain-specific) and Office-Home whose target domain is photo (which likely contains domain-specific). We observe that Rotated MNIST still remains stable at around 98% on average while that of Office-Home decreases by about 5.4%. 
Thank you for suggesting, this continues to prove the effectiveness of domain-specific features.

This paper studies the Domain Generalization (DG) task in which multiple source domains are observed and target domains are assumed to be unknown. The authors proposed a new meta-Domain Specific-Domain Invariant (mDSDI), going beyond the invariance view to highlight the Domain Specific information. The key contribution is to disentangle features in the latent space and jointly learning both domain invariant and domain specific features jointly.
The theoretical analysis is proposed for the method. Extensive experimental results have been conducted on a few datasets.     Originality:
The domain specific assumption is not new to the domain generalisation task. The novelty of the proposed meta-training is not clearly indicated. Here are a few comments. 
1. There are a lot of the definitions and assumptions, which can be shortened since it is quite standard for the domain generalisation task. 
2. No supplementary material is provided, and hence cannot check about the correctness about the proposed Lemmas and Theorem.
3. The Ablation Study has been conducted in ColoredMNIST, which is good but the authors may need either compare all the baseline algorithms on this dataset, or use one of the 5 benchmark datasets.

One more concern is about the clarity the paper and the presentation:
1. The writing needs to be improved. 
For example, the authors defined be abbreviations at the very beginning, such as  Domain Invariant (DI) and  Domain Specific (DS). After that, the abbreviations are used throughout the paper. However, sometimes it is quite challenging to memorize these abbreviations. My suggestion is that the authors can still use Domain Invariant and Domain Specific terms in the main paper.
2. Some of the reference papers are not in the standard citation format.
For example: 
[42] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[43] Seonguk Seo, Yumin Suh, Dongwan Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. CoRR, abs/1907.04275, 2019.

[42] has been published in ICLR while [43] has been published in ECCV. 
I suggest the authors to carefully check all the references, and cite them in a proper way. NA

The paper studies the problem of Domain Generalization (DG), where a model trained on single or multiple source distributions / domains is expected to generalize to data from unseen / novel distributions at test-time. The authors build on top of the observation that a subset of prior approaches rely on learning representations utilizing domain-invariant information and tend to ignore domain-specific information (which correlates with task at hand) and that ignoring domain-specific information could unintentionally harm generalization performance. Given this, the authors formalize the above concepts of invariance and specificity under a probabilistic framework and propose a practical realization of the same named meta Domain-Specific Domain-Invariant (mDSDI) representation learning. The paper claims that mDSDI is capable of explicitly disentangling domain specific and domain invariant information. In practice, key components of mDSDI include a domain-invariant encoder (trained to fool a domain discriminator), a domain-specific encoder (trained to aid a domain classifier) and a classifier (for the downstream task) that takes representations from both encoders as input to predict the semantic category. Additionally, mDSDI encourages minimizing the covariance of domain invariant and specific representations and proposes using intra-domain meta-learning to ensure representations are capable of utilizing information from source domains in the target. On the DomainBed benchmark, mDSDI outperforms 14 algorithms on 5 datasets. Additionally, the paper claims that mDSDI retains domain invariant information and captures useful domain-specific information.  I will now highlight the strengths and weaknesses of the submission, which will address quality, clarity, originality and significance of the submission.

With the exception of the points covered under weaknesses, I believe the following are the strengths of the submission in its current form:

- The paper is generally well-written and easy to follow. With the exception of a few points under weaknesses, the authors generally do a good job of motivating different components of the proposed approach -- formalizing domain invariance, specificity and label-correlated domain specificity, and the subsequent domain specific and domain invariant encoders. I generally like that the authors chose to formalize the ideas of domain specificity and invariance from an information bottleneck perspective. mDSDI being grounded in the proposed formalism is definitely a positive point in support of the paper.

- While adversarial techniques to learn domain invariant representations have been explored in the past (see [A]), the novelty of the proposed approach lies in learning an additional domain specific encoder, minimizing covariance between domain specific and domain invariant representations and additional meta learning on domain-specific features to encourage more generalizable representations. The added novelty in terms of modeling (which is grounded in the proposed formalism) is a positive point in support of the paper.

- mDSDI seems to be either competitive with or outperform prior approaches on the benchmarks (5 datasets under DomainBed). Additionally, feature visualizations in Section 3.3 indicate that domain-specific components are more capable of separating out semantic categories as opposed to domain-invariant components, providing support to the claim that relying on domain-specific information is beneficial. I also like the additional ablation highlighting the utility of individual design choices in a controlled colored-MNIST setting. Although on a controlled and simplistic benchmark, this ablation is still useful for future work building on top of the proposed approach. 

[A] - Domain Generalization with Adversarial Feature Learning

I don’t have major weaknesses to point out about the draft. The following comments / concerns are centered mostly around some clarity issues and lack of some necessary experimental comparisons / ablations.

- A minor clarification question. In the meta-training for domain-specific information setup, the authors state in L207-208 that each source domain is split into two sub-domains. This is slightly confusing Does this mean instances from domain i in a minibatch are split into two groups? Or that k-source domains in a minibatch are split into k-1 meta-train and 1 meta-test domains? The algorithm seems to imply the latter but the text seems to imply something else. Could the authors clarify if it’s the latter?

- The authors introduce the additional covariance minimization step between the domain invariant and specific representations to encourage disentanglement. While I appreciate the extensive ablations conducted by the authors to highlight the contribution of several design choices in Table 2 of the main paper (and in supplementary), the paper would benefit if the authors include additional ablations demonstrating the utility of the disentanglement loss (to what extent is performance impacted in the presence and absence of covariance minimization). This would help understand to what extent is incentivizing explicit disentanglement necessary on the DomainBed datasets and colored-MNIST. I would encourage the authors to include such an ablation.

- If I understand correctly, learning domain specific and domain invariant representations in mDSDI and the obtained results are somewhat implicitly conditioned on the availability of domain labels -- instances belong to different domains because the domain labels say so. The experiments and ablations in the paper highlight the utility of domain specific representations and the underlying utility (even under the formalism) is conditioned on the availability of domain labels. While this is a perfectly fine assumption, an additional experiment supporting the claim of learning useful domain specific representations can further be strengthened by an experiment where domain labels are randomized -- as in, say if there are 4 source domains and 1 target domain, data at training time is grouped randomly into 4 bins instead of by domain. If the out-of-domain results obtained are worse, then there’s additional validation for the utility of domain-specific representations. If they’re not significantly different, then the implication is that domain-specific representations are likely not required. I think including such an experiment is beneficial for the underlying claims of proposed formalism and will likely improve the current paper. I would encourage the authors to conduct this experiment both for one of 5 datasets under DomainBed and the controlled colored-MNIST setting.  
 Overall, I generally like the paper and the formalism adopted by the authors to ground domain specific and domain invariant components. I don't have major limitations to point out beyond the ones highlighted under weaknesses. I think addressing the highlighted experimental concerns are critical to the proposed approach. Barring those points, the paper is generally well written and the authors do a good job of discussing scenarios where mDSDI tends to help for several datasets (in the main paper and in supplementary).

I don't see any potential negative social impacts of the work.

The paper proposes meta-Domain Specific-Domain Invariant method to jointly learn disentangled domain specific and domain invariant features for domain generalisation. The usefulness of domain specific features is analysed under information theory. Experiments on several benchmarks show the effectiveness of the method.  Originality: The analysis of the usefulness of domain specific features in domain generalization under information theory is interesting. However, the disentanglement of domain invariant and domain specific features have been proposed in domain generalization [1]. The techniques used in this paper, such as Meta-learning, adversarial learning for invariant features, are also well-studied in many other works [2, 3, 4], which are not cited in the paper.

Quality: The paper proposes to exploit both domain specific and domain invariant features for domain generalization and provides analysis under information theory, which is interesting. However, the detailed implementation and experiments have little to do with the information theory, making the model and theoretical analysis a bit disjoint. There are also some further experiments needed to support the method. See limitations.

Clarity: The paper is clearly written. The implementations and experiments seem a bit disjoint with the theoretical analysis. See above.

Significance: The paper proposes an interesting theoretical analysis about the relationship between domain specific features and domain invariant features in domain generalization. 

[1] X Peng, Z Huang, X Sun, and K Saenko. Domain Agnostic Learning with Disentangled Representations. In International Conference on Machine Learning, 2019.
[2] Q. Dou, D. C. Castro, K. Kamnitsas, and B. Glocker. Domain generalization via model-agnostic learning of semantic features. In Advances in Neural Information Processing Systems, 2019.
[3] S. Zhao, M. Gong, T. Liu, H. Fu, and D. Tao.  Domain generalization via entropy regularization.  In Advances in Neural Information Processing Systems, volume 33, 2020.
[4] Y. Balaji, S. Sankaranarayanan, and R. Chellappa. MetaReg:  Towards domain generalization using meta-regularization. In Advances in Neural Information Processing Systems, volume 31, pages 998–1008,2018.

Additional comments:

   (1)	The method is proposed based on information theory but there is little modules or supports about the information theory in the implementation and experiments, making the method not so convincing.

   (2)	The technical novelty seems not so strong because adversarial learning and meta-learning used in the methods are both widely studied in domain generalization.

   (3)	The paper misses some important references, such as the references provided in Main Review.

   (4)	The paper proposes to jointly learn domain invariant and domain specific features. A question is why the domain invariant and specific features need to be disentangled first and then aggregated for prediction. How about jointly learn the features without disentanglement and aggregation? More analysis and supportive experiments are needed.

   (5)	The difference of the extracted domain invariant ZI and domain specific features ZS is not clear. What will happen if the LD term in Eq. (8) was removed? The other hyperparamters lambda in Eq. (8) also need ablations.
 NA

Meta-review: This is a borderline paper. 

Though the theoretical analysis looks nice, it may not be useful for the domain generalization setting. Specifically, the key theoretical result (Theorem 1) suggests that the representation learned by incorporating both domain-invariant and domain-specific information has a stronger dependence on labels than that learned by only considering domain-invariant information. However, there is no guarantee that such representation can generalize well to the "unseen" target domain(s). The theoretical results are based on some assumptions on the domain-invariant and domain-specific feature extractors. However, in the setting of domain generalization, as no target domain data is accessible, the domain-invariant feature may be "invariant" to the source domains only, and the domain-specific features of the target domain are not able to be extracted (accurately). This means that the theoretical results are built based on a very ideal case, but not practical for the domain generalization setting. It would make more sense if this work is applied to the domain adaptation setting where unlabeled data and some labeled data from the target domain are available for training (semi-supervised domain adaptation). Alternatively, the authors need to analyze under what conditions or assumptions for the unseen target domain, the proposed method can generalize well.

Moreover, it seems that the theoretical results are borrowed from existing findings on multi-view learning [14,16]. Is the extension of the theoretical findings from the multi-view setting to the multi-domain setting straightforward? It would be more clear if the authors could highlight their theoretical contributions compared with the techniques used in multi-view learning.

By considering the above comments, I recommend rejection for this work. 


Decision: accept