Variational Invariant Learning for Bayesian Domain Generalization

Abstract: Domain generalization addresses the out-of-distribution problem, which is challenging due to the domain shift and the uncertainty caused by the inaccessibility to data from the target domains. In this paper, we propose variational invariant learning, a probabilistic inference framework that jointly models domain invariance and uncertainty. We introduce variational Bayesian approximation into both the feature representation and classifier layers to facilitate invariant learning for better generalization across domains. In the probabilistic modeling framework, we introduce a domain-invariant principle to explore invariance across domains in a unified way. We incorporate the principle into the variational Bayesian layers in neural networks, achieving domain-invariant representations and classifier. We empirically demonstrate the effectiveness of our proposal on four widely used cross-domain visual recognition benchmarks. Ablation studies demonstrate the benefits of our proposal and on all benchmarks our variational invariant learning consistently delivers state-of-the-art performance. 

Reviews and author responses:

We thank AnonReviewer3 for acknowledging our methodology, definitions and ablation study are clear and well presented.

We agree our claim on being the first Bayesian approach to domain generalization is not precise enough. We have softened in Section 1: ''We adopt Bayesian neural networks to domain generalization'', in Section 3 '' the approach based on Bayesian neural networks has not yet been explored in domain generalization'' and in Section 5: ''We introduce Bayesian neural networks into the model''. We have also cited Louizos et al. (2016) in Section 3. Thank you.

We thank AnonReviewer2 for acknowledging the importance of uncertainty in domain generalization as well as the effectiveness of our proposal.

1. We clarify our novelty w.r.t. Motiian et al. (2017) in Related Work: ''Similar to our proposal, CCSA by  Motiian et al. (2017) also aligns representations across domains in the same class.
Specifically, CCSA utilizes an L2 distance between deterministic features while we exploit Bayesian neural networks to learn domain-invariant representations by minimizing the distance between domain distributions. 
Theoretically, minimizing the distance between distributions incorporates larger inter-class variance than minimizing distance of deterministic features.
Moreover, we apply our variational invariant learning to both the feature extractor and the classifier, while CCSA only considers an alignment loss on the feature representations.'' 
On Office-Home our proposal results in better domain generalization than CCSA for all domains (Table 4). Thank you.

2. We provide the suggested hyperparameter ablation in Appendix C. The optimal values of $\lambda_{\boldsymbol{\phi}}$, $\lambda_{\boldsymbol{\psi}}$ and $\pi$ are 1, 100 and 0.5 respectively. The performance is not very sensitive to changes of the values of $\lambda_{\boldsymbol{\phi}}$ and $\lambda_{\boldsymbol{\psi}}$ when they are in a reasonable range. $\pi {=} 0.5$ is also commonly observed in other works. More analysis is provided in Appendix C.

Minor comments
 
1. The domain-transform function $g_{\zeta}(\cdot)$ is defined as a mapping function that is able to project any sample $\mathbf{x}_{{s}}$ from domain $\mathcal{D}_s$ to other different domains $\mathcal{D}_\zeta$. Usually the exact form of the function is not necessarily known.
We have clarified this in Section 2.2.

2. We tried to enhance dissimilarity of different classes across domains by maximizing the Wasserstein distance between features, but we did not notice any notable performance improvement. We leave further exploration to future work. 

3. We clarify in the the updated manuscript that $\mathbf{z}_s$ and $\mathbf{z}_t$ in Eq. (10) correspond to samples of the feature distributions not the means.


We thank AnonReviewer4 for acknowledging our idea is natural and computationally practical.

1. We clarify in Section 4.1: ''We choose $\lambda_{\boldsymbol{\phi}}$ and $\lambda_{\boldsymbol{\psi}}$ based on the performance on the validation set and their influence is summarized in the new Fig 3 in Appendix C.''
Indeed, this could be interpreted from a MLE perspective.

2. We clarify in Section 4.3: ''The baselines on PACS (Table 3), Office-Home (Table 4), and rotated MNIST and Fashion-MNIST (Table 5) are all based on the same vanilla deep convolutional ResNet-18 network, without any Bayesian treatment, the same as row (a) in Table 1.''

We thank AnonReviewer1 for acknowledging our method is novel and promising. 

1. We have added the requested ablation study about the number of feature extraction layers on the PACS dataset in Section 4.2, Table 2. 
When introducing another Bayesian learning layer $\boldsymbol{\phi} '$ without the domain-invariant property into the model, as shown in the second row of Table 2, the average performance improves slightly. 
If we introduce both the Bayesian learning and domain-invariant learning into $\boldsymbol{\phi} '$, as shown in the third row, the overall performance declines a bit. 
One reason might be the information loss in feature representations caused by the excessive use of domain-invariant learning.
For the efficiency of the model, due to the Bayesian inference and Monte-Carlo sampling, more variational-invariant layers leads to higher memory usage and more computations, which is also one reason for us to apply the variational invariant learning only to the last feature extraction layer and the classifier.

2. We clarify in Section 4.2: '' The feature distributions $p(\mathbf{z}|\mathbf{x})$ are unknown without Bayesian formalism, leading to an intractable $\mathcal{L}_{I}(\boldsymbol{\phi})$. Therefore, we do not conduct the experiment with only the domain-invariant loss on both the classifier and the feature extractor.''

3. We add to Section 4.3: '' L2A-OT and DSON outperform the proposed model on some domains of PACS and Office-Home. L2A-OT learns a generator to synthesize data from pseudo-novel domains to augment the source domains. 
The pseudo-novel domains often have similar characteristics with the source data. Thus, when the target data also have similar characteristics with the source domains this pays off as the pseudo domains are more likely to cover the target domain, such as ''Product'' and ''Real World'' in Office-Home and ''Photo'' in PACS. When the test domain is different from all of the training domains the performance suffers,  e.g., ''Clipart'' in Office-Home and ''Sketch'' in PACS. Our method generates domain-invariant representations and classifiers, resulting in competitive results across all domains and overall. 
DSON mixtures batch and instance normalization for domain generalization. This tactic is effective on PACS, but less competitive on Office-Home. We attribute this to the larger number of categories on Office-Home, where instance normalization is known to make features less discriminative with respect to object categories (Seo et al., 2019).
Our domain-invariant network makes feature distributions and predictive distributions similar across domains, resulting in good performance on both PACS and Office-Home.''

4. Selecting training samples by predictive uncertainty is interesting and we do not set a threshold in our method to do so. Our Bayesian framework captures and models uncertainty to alleviate the overconfidence on out-of-distribution test data.

Summary:
The paper proposes variational invariant learning (VIL) as a framework for probabilistic inference that jointly models domain invariance and uncertainty. The approach exploits variational Bayesian approximation in both feature encoding and classifier layers in order to facilitate domain generalization and invariance. The paper evaluates VIL on benchmarks for cross-domain visual object classification.

Positives:
The paper presents their methodology and definitions very clearly. Comparisons to state-of-the-art methods are well presented. The ablation study is very clear to provide a thorough understanding of the approach. Overall I like this paper.

Concerns:
The authors state: "the Bayesian approach has not yet been explored in domain generalization" or "this is the first work to adopt variational Bayes to domain generalization". I may not strongly agree on these statements in their current form. There has been some previous work on extending variational autoencoding frameworks for domain generalization from several perspectives (e.g, adversarial inference). This type of variational inference based models ideally exploit a Bayesian approach for domain generalization as well. I would expect the authors to rephrase their claims in a more specific manner to make this distinction. Regarding this concern, one example to consider for instance is "The Variational Fair Autoencoder" (Louizos et al., ICLR 2016) approach for invariant representation learning with variational autoencoders for domain generalization.

Summary:
- The paper proposes a Bayesian inference framework for domain generalization (DG) that deals with both the uncertainty and domain shit. 
The proposed method treats the uncertainty effectively by applying the variational Bayesian inference to the last two layers of the model (feature representations and classifier layers).
To make the model domain-invariant, the proposed method introduces a domain-invariant principle that makes distributions of the model's outputs (representations) in the same class similar across domains. 
The paper experimentally demonstrated the effectiveness of the method with four visual datasets.

Pros:
- The problem tackled by this paper is very important because accurate prediction is generally difficult in DG due to the lack of target training data and therefore considering the uncertainty is especially important.
- Experimental results with four widely used datasets show the effectiveness of the method.

Cons:
-  The technical novelty of this method seems a bit low. This is because this method is a relatively simple combination of Bayesian modeling of neural networks and some existing techniques for DG. 
In previous works for DG, techniques to bring distributions of model's outputs in the same class closer together across domains have been proposed.
For example, CCSA [1] learns the domain-invariant representations by matching the representations in the same class across domains.

[1] Motiian et al., Unified deep supervised domain adaptation and generalization, ICCV2017

- More analysis of hyperparameters such as $\lambda_{\phi}$, $\lambda_{\psi}$, and $\pi$ would improve the quality of this paper.

Reasons for Scores:
- Although I like the research direction of this paper as described in Pros, I have some concerns about the technical novelty of this method as described in Cons.

Minor comments: 
- definition 2.1 (domain invariance) is a little confusing for me although I can see what the method wants to do by seeing Eq. (7). For example, what is the formal definition of domain-transform function?
- Although the loss function for domain-invariant learning (7) seems to bring the distributions of the same class across domains closer together, can the method incorporate a
mechanism to enhance dissimilarity of the different classes across domains? I think that it might improve performance.
- There are some ambiguous statements. In Eq. (10), are $z_{s}$ and $z_{t}$ the means of distribution of $q(z|x,\phi)$?


The authors propose a variational Bayesian approach to domain adaptation.  The goal is to achieve flexibility in specifying domain invariance as well as modeling uncertainty. The variational approach could also help generalization.

The key idea is simple and straightforward — a distribution over domains is first specified and variational approximation is then used for parameter estimation. In particular, the authors decompose the variational distribution in terms of the domain and classifier parameters, applying the approximation to the last two network layers. The objective function is given as a weighted sum of the KL divergence corresponding the two component distributions.

In the empirical evaluation, the authors compare the performance on in-distribution and out-of-distribution data. They conclude that the proposed method achieves generally improved accuracy on several image datasets, including PACS, Office-Home, as well as MNIST variants. 

Strengths:
- The presentation is clear. The proposed Bayesian formulation is natural and the resulting learning problem is computationally practical.

Weaknesses: 
- Some parameters such as the weights \lambda_{\phi} and \lambda_{\phi} do not have clear Bayesian interpretations.
- In the experimental evaluation, different baseline methods are used for each datasets. 

Questions:
How do the parameters \lambda_{\phi} and \lambda_{\phi} impact the performance? Could we interpret these parameters from the MLE perspective?

########################

Summary:

The paper introduces a variational invariant learning approach with Bayesian approximation for domain generalization.

####################

Reason for score:

Overall, the paper is above the borderline. I like the idea of utilizing the Bayesian variational learning to address the domain generalization problem, which is very novel and promising. My major concern is about some unclear parts described in the paper and insufficient experimental comparison (see cons below). Hopefully, it would be grateful that the authors could address my concerns during the rebuttal period.

########################

Pros:

(1) The proposed variational Bayesian learning framework in the paper to represent uncertainty and enhance the generalization is reasonable and interesting, which can inspire other researchers in related fields.

(2) The introduced domain-invariant principle to establish a domain-invariant feature extractor and classifier seems promising, which can lead to an end-to-end framework with CNN and a Bayesian network.

(3) Extensive experimental results on four domain generalization benchmarks show the proposed method obtains a new state-of-the-art performance, which is appealing and convincing.

########################

Cons:

(1) In the paper, the authors claim that they only apply the domain invariance to the last feature extraction layer. In order to examine the relevance between the number of layers and the performance, an ablation study about the number of feature extraction layers should be added. In addition, the efficiency discussion about the model layers will be better to denote the setting chosen in the work.

(2) In Table 1, what happens if without Bayesian and with both invariant, i.e., only introducing invariant loss into both classifier and the feature extractor? I notice that such a certain situation is missing in the experiments.

(3) As shown in Table 2 and Table 3, DSON and L2A-OT can outperform the proposed model in terms of certain metrics, it would be better if the authors can analyze the results and give some comparisons about that.

(4) As for the uncertainty estimated by the proposed Bayesian network, is there any threshold being set in the model to determine which samples can be used in the training？ Moreover, how to utilized such estimated uncertainty should be discussed further.

########################

Questions during the rebuttal period: 

Please address and clarify the cons above. Thank you!


Meta-review: Although the proposed method shows sota results, it is a simple combination of two existing methods, a bit of Bayesian + domain generalization.  It seems that the total improvement by the proposed method is just the sum of improvements by Bayesian and by domain generalization.  No synergy between Bayesian and domain generalization is observed.

I personally doubt that the Bayesian treatment of the domain generalization loss is not essential.
The derivation in Sec. 2 is unnecessarily complicated.  In derivation from eq(3) to eq(5), the authors first "extend" (3) to (4), which is not appropriate ((4) can hold even if $p(y_{\zeta}|x_{\zeta})$ is highly diverse. ).  After that the authors apply Jensen to come back to an appropriate form (5), which is a weighted sum of distances (which is an appropriate criterion).
If they start from Eq.(5), the proposed objective is simply the sum of the standard ELBO (2) and a natural domain invariance loss (5).  For non-Bayesian treatment of the domain invariant loss, you could simply replace the KL by Lp norm between $y_s$ and $y_\zeta$.  I expected that by answering to the question by Reviewer 1 the authors would prove synergy between Bayesian and domain generalization.  But the authors just excused that
'' The feature distributions  are unknown without Bayesian formalism, leading to an intractable $L_I$. Therefore, we do not conduct the experiment with only the domain-invariant loss on both the classifier and the feature extractor.''
I don't really understand what the authors mean, but the authors should have explained why you cannot replace the KL with non-Bayesian Lp loss.




Decision: withdrawn-rejected-submissions