A Rate-Distortion Approach to Domain Generalization
Authors: Yihang Chen
Grigorios Chrysos
Volkan Cevher
Venue: ICLR 2022 Submitted (2022)

Abstract: Domain generalization deals with the difference in the distribution between the training and testing datasets, i.e., the domain shift problem, by extracting domain-invariant features. In this paper, we propose an information-theoretic approach for domain generalization. We first establish the domain transformation model, mapping a domain-free latent image into a domain. Then, we cast the domain generalization as a rate-distortion problem, and use the information bottleneck penalty to measure how well the domain-free latent image is reconstructed from a compressed representation of a domain-specific image compared to its direct prediction from the domain-specific image itself. We prove that the information bottleneck penalty guarantees that domain-invariant features can be learned. Lastly, we draw links of our proposed method with self-supervised contrastive learning without negative data pairs. Our empirical study on two different tasks verifies the improvement over recent baselines. 


Under review as a conference paper at ICLR 2022
A RATE-DISTORTION APPROACH TO DOMAIN GENER- ALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Domain generalization deals with the difference in the distribution between the training and testing datasets, i.e., the domain shift problem. A principled approach to domain generalization is by extracting domain-invariant features. In this paper, we propose an information-theorriic approach for domain generalization. We first establish the domain transformation model, mapping domain-free latent image into domain. Then, we cast the domain generalization as a rate-distortion problem, and use the information bottleneck penalty to measure how well the domain -enein--rree latent image is reconstructed from a compressed representation of domain-specific image compared to its direct prediction from the domain-specific image itself. We prove that the information bottleneck penalty guarantees that nomain-invarian features can be learned. Lastly, draw links of our proposed method with self- supervised contrastive learning without negative data pairs. Our empirical study on two different tasks verifies the improvement over recent baselines.
INTRODUCTION
Deep neural networks (DNNs) are highly expressive models that reach state-of-the-art performance in challenging tasks, such as speech and visual recognition (Devlin al., 2018; He et al., 2016), by capturing complex correlations among input elements, e.g., pixels of an image. However, the correlations might also contain spurious features that hurt the generalization performance of DNNs on diito---iitriittionn samples (Szegedy et al., 2013; Beery et al., 2018; 8; Alcom et al., 2019). Un- fortunately, real-world applications often encounter such out-r--disttibution samples, e.g., when the training domain does not match the testing domain. A prominent example is deblurring, where models are trained on simulated blurs which differ substantially to real-world blurring (Koh et al., 2021). In other words, generalization across domains is critical task before deploying DNNs to real-world application.
Learning features that are invariant across multiple training domains, and using those features for dut-of-disttibuutin generalization has emerged as a significant topic in domain generalization. In domain generalization, multiple source domains are accessible during training, but the target domains are not (Blanchard et al., 2011; 1; Muandet et al., 2013). Invariant risk minimization (IRM) ( (Arjovsky al., 2019) is prominent approach for learning domain invariant features. However, IRM suffers from the case when the invariant features contains full information about the label (Ahuja et al., 2020). To deal with this shortcoming, Ahuja al. (2021) introduce the information bottleneck theory on neural networks (Tishby & Zaslavsky, 2015), and show that their method will be guaranteed to converge the invariant features. On the empirical side, series of works align source domain distributions for domain-invariant representation learning by either direct construct auxiliary penalty (Duan et al., 2012: Sun & Saenko, 2016; Li et al., 2018b:c; 2017; Niu al., 2015), or meta learning (Li et al., 2019; Balaji et al., 2018: Li et al., 2018a).
There are also series of work do not rely on invariant features. They can be categorized as (1) domain- specific method: Domain2 Vec (D2V) (Deshmukh et al., 2018) learns domain-speciio DMG (Chattopadhyay et al., 2020) aims to learn domain specific masks; and (2) augmentation method: (Volpi al., 2018) augments the dataset adversarially. L2A-OT (Zhou et al., 2020) augments data with image information. Despite their success, there is no guarantee that empirical methods can solve the task across different environments.
embedding,

---

Under review as a conference paper at ICLR 2022
In this paper, an information -ieprmmattoo-thorttiic approach to tackle domain generalization. We assume there is domain femeaie-freeeeee latent instance (e.g., an image) that captures the invariant features we want to extract. We define domain transformation model that maps the domain-free latent instance into domain and then we apply the rate distortion theory to obtain a domain-invariant representation. The proposed method, called Twins, is guaranteed converge the invariant feature under the linear classification structural equation model (Ahuja et al., 2021). We evaluate our method on linear unit tests (Aubin et al., 2021) and variants of MNIST dataset (LeCun & Cortes, 2010; Xiao al., 2017; Clanuwat et al., 2018), which validates the theoretical analysis and demonstrates how the proposed method can outperform the previous ones. Our contributions can be summarized as follows:
We cast domain generalization rate distortion problem and prove how the proposed method can converge.
We illustrate how the proposed method extends previous results on domain generalization, and draw links to self-supervised contrastive learning. We demystify the success of contrastive learning by giving a contrastive learning based domain generalization algorithm with thcoretical guarantee.
We evaluate our method on two datasets and observe consistent improvement over existing base- lines.
PRELIMINARY ON DOMAIN GENERALIZZATON
Assume that the instance-label pair (X, Y) is sampled from an unknown distribution P(X, Y). The objective of standard supervised learning is to learn predictor f that is able to predict the labels Y of corresponding instances X for each (X, Y) ~ P(X, Y), given the finite training samples drawn from the underlying distribution P(X,Y) Y..
Unlike the standard supervised learning tasks, in domain generalization, we cannot sample directly from the distribution P(X, Y). Instead, we only observe (X, Y) under different domains E Eat. denoted as (X*, Yo) ~ P (X*, y²). We also assume that e E Eall is distributed as e ~ Pe. Given samples from finite subset Euran Eall of all the domains, the goal of the domain generalization problem is to learn a predictor / that generalizes across all possible domains. This can be summarized as follows:
Problem 2.1 (Domain generalization). Let Evain & Eall be finite subset of training domains. We have access to the data for each training domain Etrain E Evain. but have no access to the data for each test domain etstt E Eail Emain Given a function class F and loss loss function É, our goal is to learn a predictor I E F using the data from the training domain such that I minimizes the worst-case risk over Eall- Define the risk of the predictor on the domain R$) = (x (x(),, Y*). We want to solve the following min-max optimization problem:
minimize
max R (f)- JEF e
(DG)
We establish the domain transformation model to characterize the relation between domain-aware instance X and the domain-invariant latent instance X in the Assumption 1, which first appears in Robey et al. (2021).
Assumption 1 (Domain transformation model). Let de denote Dirac distribution for Eail- We assume that there exists a measurable function G X x Eall + X, which we refer to as a domain transformation model, that parameterizes the inter-domain covariate shift via
1 (x) -a G # (P(X) 8e)
We E Eall.
where # denotes the fush-forward measure and =d denotes equality in distribution.
The Assumption 1 can somewhat reflect the generation of domain specific instances. For example, the multiple different views of a 3D object (Niu et al., 2015), different angles of the image (Rotated MNIST (Worall et al., 2017)). Besides, the MUNIT architecture (Huang et al., 2018) can effectively distangle the domain-free latent instance X and the specific environment e, and thus can be used as the domain transformation model G (Robey et al., 2021).
Let & denote the feature representation mapping, w denote the classifier and w o denote the full predictor. The regret of the network on the domain is denoted as Rt (w +).
2

---

Under review as a conference paper at ICLR 2022
Next, we define standard properties related to the datasets used in the domain generalization literature (Ahuja et al., 2021). For each e E Ealls the distribution (X) ~ IP satisfies the following properties: (1) a map 4*, which we call an invariant feature map, such that [E[ ( (5x(x)) same for all e E Eall and Y I & (xx"), where - means mutual independence. (2) 3 map Vr* which we call spurious feature map, such that E[ye v/* (X+)] is the same for all E &all and
the
Y L / (Xx") for some domains. & often hinders learning predictors that only rely t. For example, in the CMNIST dataset, the d extracts the underlying digit and y extracts background color.
The baseline algorithm for domain generalization Equation (DG) is the Empirical Risk Minimization, Le. directly minimizing the empirical risk the training domains:
(e b),
Cirin
eEEiii
where &vainl denotes the number of training domains.
We say that a data representation & elicits an invariant predictor across the set of training domains Evain if there a predictor w that simultaneously achieves the minimum risk, i.c. E arg min R 4). Ve E Euain- Using this notation, the main objective of Invariant Risk Minimization (IRM) is stated as:
min 12,0
Σ eEEmm
(w o ②),
s.t. w E arg min R2 (w'' o 2), Ve E (3)
Euriia
w/
Lastly, we rely on the notion of 'informativeness' about the datasets (Ahuja et al., 2021). There are two such categories of informativeness. In the first case, the invariant features & (X*) are partially informative about the label, i.e. Y y x=| (X*), ), and color contains information about label not contained in the uncolored digit. In the second case, invariant features are fully informative about the label, i.., Y x (X'), i... they contain all the information about the label that is contained in input X Many real-world image datasets have fully informative invariant features, the labels are deterministic function of the nomain-invarian features and domain-aware spurious features do not affect the label.
METHOD
3.1 RATE DISTORTION & INFORMATION BOTTLENECK PRINCIPLE
Given a domain-free free latent instance X E dx its observation in a domain e is denoted as X := G(X,e), e). We want learn the feature Z ((")) Rd which is informative about the domain- free variable X, but invariant (i.e. uninformative) to the specific domain e. We use rate inte-distortion theory (Davisson, 1972; Blau & Michaeli, 2019) to formulate domain generalization problem.
Rate-distortion theory is a major branch of information theory which provides the theoretical foundations for lossy data compression. An encoder & encodes domain-aware instances Xe. We want the representation Z = (X*) to be domain-invarian,, so we feed Z into decoder which outputs domain-invariant X. We minimize the distortion between the original domain-aware instance X" and the reconstructed domain-free instance X. The distortion function measures how well X predicted from a compressed representation 2 compared to its direct prediction from X", This trade-off is captured by the following loss function:
(i e) = EX-Px.e<2. 1(2"; x²) - (I( IB objective
(4)
where denotes the mutual information, is the parameter of the representation function 4 and is constant.
In the following, we consider two cases: discrete and continuous variables, owing to their different definition of entropy.
Discrete case: Since the representation function deterministic with respect to 8, we can rewrite Equation (4) through classical identity of mutual information: T(X;Y) H(X) H(X(Y),
3

---

Under review as a conference paper at ICLR 2022
where H denotes the Shannon entropy for discrete variables, follows:
eI Exx,e X*) (I((Z;
[xx,H((2*) - -(H(2))- H(Z))))
(5)
1-
Ex (Z9)
3 H(?)),
where in the last equality we omit the overall scaling factor of the loss function.
I S B 1, since H.)) bounded below by O, setting + to be constant will clearly minimize the
penalty, which is uninformative about the representations we want to learn. Hence, we set B > 1, and replace 1- with ーλ, where 0 S X < 1. The IB objective can be rewritten B
LI = ex -Z"
(6)
Continuous case: In terms of continuous variables, the differential entropy h(.) not bounded below, which hinders analysis. To overcome this, define the lower bounded differential entropy h(X) h(X + E). where e is the independent bounded zero-entropy noise Uniform(O, 1).
Thus, Xx( 2 M(E) = O. We can replace the Shannon entropy (-) with lower bounded differential
entropy 7(-) in Equation (6): Lin(0, e) Ex Ex,. - AZ(Zz))
(7)
For simplicity, we define H be the Shannon entropy for the discrete variables, or lower bounded entropy h for continuous variables in the main text. We define (f) == Ex H((()))) We can extend the Empirical Risk Minimization (ERM) algorithm include the IB Penalty, and the resulting algorithm, denoted as Twins-ERM method, is the following:
min U.. Jb
Σ (x XH (Φ)
S.I.
Σ cEEmm
R () Sr
(8)
Errain
CEEin
where is the threshold on the empirical risk on the training domains.
In addition ERM, another popular minimization framework invariant risk minimization (Ar- jovsky et al., 2019). The proposed penalty can be readily incorporated into the IRM framework, we call the resulting algorithm Twins-IRM:
min 10
Σ
H H² (1).
Emit
S.t.
Σ
( 4) Sr,
arg min H (w 4), Ve Erain. "
Eyain
egEnan
3.2 THEORETICAL GUARANTEE
In this subsection, we establish the theoretical guarantee of our algorithm under the linear classification case. We consider the following standard 0-1 classification model literature (Ahuja al., 2020; 2021):
Assumption 2. Linear classification structural equation model. In each e E Eail-
1 2 Ne (Xinv; X
ye +- T(winn Xinv) E Ne, No Bernoulli(q.4s
(10)
xe S(Xe xe x X + G(Xinvee))
where 1(x) is 1 if is positive else O, wiy E 18mh with | = is the labelling hyperplane, S () (m+t)) and G is continuous domain transformation.
. X E Rm R E R..
Before presenting our main theorem, we first add two assumptions on the support of invariant features. Define the support of the invariant features Xe in environment X
4

---

Under review as a conference paper at ICLR 2022
Assumption (Invariant feature support overlap). The union of support of the invariant features of the training domains covers support of the invariant features of all the domains. i.e. UeEcar & S
UeEEom Xinv
Assumption 4 (Strictly separable invariant features). The training support of invariant fea- tures UsEEma Kin is strictly separated by the labelling hyperplane Waye In other words, minzeU.ee.... X (g .z) (Winv <>>0 0.
Assumption and describes the property of the support of invariant feature. Under these assump- tions, we propose our first main theorem:
Theorem 3.1. Suppose each E Eall follows Assumption 2, and Assumptions and hold for the invariani features. Also, for each E Emain assume that X AX iny + W where A E Boxm W E Io is continuous, bounded, and zero mean noise. Each solution to Twins-ERM and Twins-IRM (Equation (8) and Equation (9), with E as 0-1 loss, and q) solves the domain generalization problem (Equation (DG)).
Sketch of Proof. The full proof is provided at Appendix We only present the main idea here. Denote @t as the solution to Equation (9),
$ X = SXX inv X spu) = @b inv X + o X spn inv + D spa -A) inv + t We
(Φ)
(11)
We will show that ++ [[Dimv + & spu A).0)s-1 can continue achieve an error of g(= r) across training domains, and have a lower information bottleneck penalty. Therefore, the optimal solution to (wins-ERM (Equation (8)) does not depend on the spurious noise W and hence solves the domain generalization problem (Equation (DG)).
is known that ERM and IRM fails under the assumption of Theorem 3.1 (Theorem 3 in (Ahuja et al., 2021)). This theorem shows that our algorithm can provably solve the linear classification structural equation model.
In real world, however, do not have direct access to the domain-free instance X. Hence, practically adopt image from another domain, denoted as X as a proxy for X in Twins-ERe and Twins-IRM (Equation (8) and Equation (9)). In other words, fixing e' E Erainn the Twins-ERM (Equation (8) can be rewritten as
min Σ (0( XH (Ø) s.t. Evainl
Σ eEEuan
(w Sr.
(12)
Emii
and the Twins-IRM (Equation (9)) can be rewritten as
min w.
Σ H (*(x))xx=))
- (4)
(13)
s.t. |Eraial
Σ eEEi
(s o 3) r, E arg min R (w o D), Ve E Euain-
We will show adopting proxy from another domain will still be guaranteed to solve the domain generalization problem (Equation (DG)).
Theorem 3.2. Suppose each e Eall follows Assumption 2, and Assumptions 3 and 4 hold for the invariant features. Also,for each E E assume that Xe AXe inv + W where A E Roxm W IR is continuous, bounded, and zero mean noise. Each solution to Twins-ERM and Twins-IRM (Equation (12) and Equation (13)), with & as 0-1 loss, and r = q) solves the domain generalization problem (Equation (DG)).
The full proof of Theorem 3.1 and 3.2 can be found at Appendix A.
3.3 GAUSSIAN BOTTLENECK
The IB obejetive (4) can be directly estimated by k-nearest-N neighbr method, as described in Ap- pendix C. However, such direct estimation requires large memory and is computationally intensive.
5

---

Under review as a conference paper at ICLR 2022
We denote the algorithm by kNN direct estimation Twins-Direct. In order to facilitate the imple- mentation of our penalty, we make the simplifying assumption that the datasets follow a Gaussian distribution (Chechik et al., 2005). Specifically, assuming X, X are jointly multivariate zero-mean Gaussian vectors with covariances Σx, Exs, and Z e Rdæ is a encoded version of X that must maintain given value of mutual information with X. We define the covariance of Ze and z X to be Zg and [x
Next, simplify the Equation (6). The entropy of a Gaussian distribution simply given by the logarithm of the determinant of its covariance function (up to a constant that we ignore). The loss function becomes:
LI (0, e) = Ex log det((Ez(x) A log det(E
(14)
Practical considerations: We reformulate our algorithm so that it resembles the contrastive learning method, i.e. Barlow Twins (Zbontar et al., 2021). The second term of the loss in Equation (14) maximizes det (e=))) Since computing the determinant of matrix is computationally intensive, adopt a proxy to minimize the Frobenius norm of the correlation matrix of Z". Since the correlation matrix is invariant to scaling, we can set the diagonal element of the correlation matrix of Ze to be Then, the second term of Equation (14) amounts to minimizing the off-diagonal term, i.e. the second term in Equation (16). This term essentially decorrelates the different dimensions of the representation and prevents these dimensions from encoding similar information.
Besides, it also easily be shown that the first term of Equation (14) minimizes the information the representation contains about the domain information has the same solution with the first term of Equation (16). This term maximizes the alignment between representations of pairs of domain-aware instances X and domain-free free instances X.
In practice, we have no access to the domain-free latent instance. For given instance X we use an instance with the same label, but from different domain, denoted X"' as surrogate for the domain-free latent X. We minimize the distance between pairs of instances from different domains. Sample {=f]s ~ 7 and {2ī' 11SBBB Ze', we concatenate them into matrix Z E Bxd2 and z RRBx2 where dz the dimension of z =g' After mean shifting every column of Z and z such that 1 Z = Iz = 0 (1 is a column vector full of 1s), the cross-correlation matrix C is defined as:
C2
1 13i.j dz.
(15)
1
and the final penalty defined as:
c(Z) = (1- + Σ (C))
(16)
However, we do not have access to domain transformation model either. We construct the contrastive instances by permuting the instances that have the same label in each iteration. In particular, we sample B instances { from training domains as row vectors, where B is the batch size. We concatenate the representation } = { into matrix Z E Bxdz, The contrastive batch 2 is constructed by permuting the rows of Z i.c. Z,, 2 (0)))= where is a permutation
of 2, B) such that the corresponding labels of as and (b) are identical. C defined in Equation (15) can be rewritten as:
Z
Z 1,0
C
i,j dz
(17)
7Z
Z
Such penalty can be readily incorporated into the ERM and IRM losses, i.c.
LTwins- ERM = LERM + μ c(Z), and LTwins-TRE IRM LIRM + p1 c(Z),
(18)
where LERM and LIRM denote the loss in the ERM and IRM respectively, and is the penalty hyperparameter.

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
In this section, conduct experimentation two benchmarks: Linear Unit Tests (in Section 4.1, Section 4.3) and DomainBed (in Section 4.2). Linear Unit Tests (Aubin et al., 2021) consists of several toy datasets evaluate algorithms for domain generalization and invariance learning, while DomainBed (Gulrajani & Lopez-Paz, 2020) is a unified testbed for evaluating domain generalization algorithms. We use the following four baselines across our experiments: ERM, IB-ERM, IRM, IB-IRM (Ahuja et al., 2021) in the synthetic data, and use ERM, IRM as baselines in real-world datasets.
4.1 LINEAR UNIT TESTS
The dataset describes six linear low-dimensional problems, named Example 1/1s, Example 2/2s and Example 3/3s, where the 's' dictates a different rotation matrix. Each example, called unit test, is designed to test different types of the precise distributions and the invariances captured by each example.
generalization, We describe in Appendix B.1
Benchmark details: We follow the same pipeline as those used in Aubin et al. (2021); Ahuja et al. (2021) for the model selection, hyperparameter selection, training, and evaluation. We set (dinv: dmpu) (5, 5). For all three examples, the models used are linear. The training loss is the square error for the regression setting (Example 1/ls), and binary cross-entropy for the classification setting (Example 2/2s, 3/3s). For the evaluation of performance on Example 1/ls, we report mean square errors and standard deviations. For the evaluation of performance Example 2/2s, Example 3/3s, we report classification errors and standard deviations.
Model training: For the Twins-ERM approach, there is an additional hyperparameter μ as- sociated with the z term in the final objective in Equation (18). We sample the fu from log 11 Uniform(- 3, -1). For each algorithm, we run random hyperparameter search for 20 trials, and average the results over 50 data seeds. We train each algorithm and hyperparameter trial on the train splits of all environments, for 10 full-batch Adam updates (Kingma & Ba, 2014). We choose the hyperparameters trial that minimizes the error on the validation splits of all environments, i.e. the train domn-n validation set evaluation procedure in (Gulrajani & Lopez-Paz, 2020).
We also implement an Oracle that contains randomized X Xapu in each iteration, such that it learns to ignore the spurious features.
#Eavs
ERM
IB-ERM
IRM
IB-IRM
Twins-ERM
Twins ERM
Twins IRM
Oracle
Examplel
3
13.36 1: 1.49
12.96 : 1.30
11.15: 0.71
11.68 1 0.90
14.62 = 1.20
14.42 = 0.36
10.42 : 0.16
Examplels
3
13.33 + 1.49
1292
1.30
11.07 = 0.68
11.74 + 1.03
14.64 = 1.22
13.25
1.49
10.45 0 0.19
Example2
3
0.42 : 0.01
0.00
: 0.00
0.45 0.00
0,00 0.00
0.00 0.00
0.00
0.00
0.00 = 0.00
Example2s
3
0.45 1 0.01
0,00 1: 0.01
0.45 1 0.01
0.06 0.12
0.00
1: 0.00
0.43 1 0.03
0.00 0.00
Example3
3
0.48 + 0.07
0.49 + 0.06
0.48 + 0.07
0.48 + 0.07
0.42 0.15
0.33 0.14
0.00 0.00
Example3s
3
0.49 + 0.06
0.49 + 0.06
0.49 + 0.07
0.49 +
0,50 0.05
0.42 + 0.11
0.00 0.00
Example2
6
0.37 :: 0.06
0,02 1 0.05
0.46 0.01
0.11
0.00::0000
0.07 0.11
0.00=0.00
Example2s
6
0.46 1 0.01
0.02 1: 0.06
0.46 1 0.01
0.45 0.10
0.00 1.001:00 0.00
0.47 1 0.00
0.00-0.00
Example3
6
0.33 + 0.18
0,26 + 0.20
0.14 0.18
0.16
0.25 +: 0.20
0.01-0.00
Example3s
6
0.36 0.19
0.27 =
0,14 0.18
0.19
0,31 0.19
0.44 : 0.06
0.01=0.00
Table 1: Comparisons on linear unit tests in terms of mean square error (regression, J) in Example 1/1s and classification error (classification, J) in Examples 2/2s and 3/3s. The highlighted result per example demonstrates the best performance. When #Envs=6, we do not report results on Example 1/1s, since even the oracle cannot obtain stable results across different data seeds.
The experimental results are reported in Table In the Example 2/2s, since the invariant feature contains full information about the label, we do observe that IB penalty in (Ahuja et al., 2021) and Twins penalty in our paper performs the best. In the Example 1/ls and 3/3s, the spurious feature contains partial information about the label, we generally find invariant risk can reduce the error in this case. We empirically verify the benefit of using the proposed penalty over the previously proposed baselines.
4.2 MNIST-TYPE DATASET
In the second benchmark we use DomainBed to experiment on MNIST-type datasets inspired by the construction of CS-CMNIST (Ahuja al., 2021) to evaluate covariate shift. In addition to the
7

---

Under review as a conference paper at ICLR 2022
origin MNIST (LeCun & Cortes, 2010), extend the benchmark to include FashionMNIST (Xiao et al., 2017) and KMNIST (Clanuwat et al., 2018), where the images of the latter two are used as drop-in replacements for MNIST images. The idea in CS-CMNIST is to associate each class with a color, and each image is assigned the color associated to its class with probability pe or a random color with probability 1 p²_ We construct three environments for this experiment: two training environments containing 20,000 data points each, one test containing 20,000 points. In the two training environments, the pe is set to 1.0 and 0.9 respectively. In the testing environment, the p" is set to 0, i.e., all the images are colored completely at random. A grid search in the range of {10-4, 10-3 10 10-2, 1001 1] used to determine the optimal penalty parameter μ. We fix the trade-off parameter À = 5 x 10 We run the experiments using 5 different seeds and report the mean and the standard deviation of the classification.
The results are reported in Table 2. We find that generally setting μl 10-2 would be the best choice. See Appendix D Notice that both the IRM and the ERM versions in each case perform similarly. The results reveal that ERM and IRM have the weakest performance. IB-ERM and IB-IRM increase the accuracy of ERM and IRM respectively, with Twins-ERM and Twins-IRM, outperforming all the compared methods.
ERM
IB IB-ERM
IRM
IB IB-IRM
Twins-ERM.
Twins-IRM
Twins-Direct Direct
MNIST
60.27 + 1.21
71.80 + 0.69
61.49 + 1.45
71.79 + 0.70
83.03 + 1.34
82.83 + 2.73
79.98-+ 0.87
FashionMNIST
50.92 : 1.20
51.74
1.12
48.41
+ 0,90
50.92
1.20
55.60
3,33
56.04 +: 1.79
55.20:1: 2.16
KMNIST
22.80
1.06
29.21 0.85
22,89 + 0.94
27.83 0.37
51.24
3.94
51.52 + 3.83
50.29 2.58
Table 2: Classification accuracy (f) MNIST-type datasets. Notice that the proposed Twins-ERM and Twins-IRM exhibit the best performance outperforming previous methods by a significant margin. Twins-Diree (See Appendix C) achieve similar performance with Twins-ERM and Twins-IRM
4.3 REAL WORLD DATASETS
In the third benchmark we use DomainBed to experiment on real world datasets: OfficeHome (Venkateswara et al., 2017), PACS (Li al., 2017). For our Twins algorithm, we ran hyperparameter search in the range of {10-4 10-3 10 100--2111011 10-1) for jr, and 20 hyperparameter seeds for remaining hyperparameters in the DomainBed suite (Gulrajani & Lopez-Paz, 2020). We run the experiments using 3 different seeds and report the mean and the standard deviation of the classification. For ERM and IRM, we directly borrow the reuslts from the original paper.
The results are reported in Table 3 and 4. We find that our Twins-IRM algorithm obtain consistent improvement over the baselines.
Algorithm
A
C
P
R
Avg
ERM
61.3 + 0.7
52.4
+ 0.3
75.8
+.1 0.1
76.6
0.3
66.5 + 0.3
IRM
58.9 + 2.3
52.2
+
1.6
72.1
+
2.9
74.0
+ 2.5
64.3 + 2.1
TWins-IRM
64.8
0.2
52.6
+ 0.7
77.5
0.2
78.9
0.3
68.5 + 0.4
Table 3: Classification accuracy (1) on OfficeHome.
Algorithm
C
P
S
Avg
ERM
84.7 + 0.4
80.8 + 0.6
97.2 + 0.3
79.3
+ 1.0
85.5 + 0.7
IRM
84.8 1.3
76.4
+ 1.1
96.7
0.6
76.1
1.0
83.5 + 1.1
Twins-IRM
88.0 + 0.3
79.6 + 0.4
97.9
+ 0.5
80.1
+ 0.9
86.4 + 0.6
Table 4: Classification accuracy (1) on PACS.
RELATED WORK
5.1 RELATION TO INVARIANT RISK MINIMIZATION
Relation to IB-ERM/IB-IRN (Ahuja et al., 2021). Ahuja et al. (2021) introduce the information bottleneck method. However, similar to Tishby et al. (2000), Ahuja et al. (2021) utilize the information
8

---

Under review as a conference paper at ICLR 2022
bottleneck to learn a representation that compresses the imput much as possible while preserving all the relevant information about the target label. Ours instead is label-free and tries to preserve the relevant information about the domain-free latent image during compression, and uses another image with the same label but different domain as surrogate for the unknown domain-free latent image. Essentially, setting = 0 will reduce our penalty into the one in Ahuja et al. (2021); but / > 1 can help eliminate the trivial solution that the representation mapping is constant, and is shown to achieve better performance on various datasets.
Previous work has demonstrated that the entropy penalty alone (i.e. 8 - 0) might fail in specific case, such as in Section 4 in Ahuja al. (2021). Nevertheless, our proposed framework does not suffer from this counter-example. We introduce such an example next as a simple classification problem. In each E Emain Y <- Xi @ N and X <- Y () Ve, where all the random variables involved are binary valued, noise N Ve are Bernoulli with parameters q (identical across Emain), et (varies across Enain) respectively. If c < q. then in Erain predictions based on Spp are better than predictions based on Xfav- If & selects Xiny. the IB pemalty equals -X ]i while +b selects X apu the IB penalty equals H(N + V*) XH(X) Since X + X ① No $ Ve, we have - -XH(X inv) < H(N @ V*) XH (X= spu) by Lemma A.2 in the Appendix. Our IB penalty is then able to select the invariance term X On the other hand, if X obeys uniform Bernoulli distribution, its entropy will be no lower than spu and hence the entropy term alone is not enough.
5.2 RELATION TO CROSS-DOMAIN COVARIANCE METHOD
Aligning the cross-domain distribution has been studied extensively in the domain generalization community both empirically and theoretically (Sun & Saenko, 2016; Li et al., 2018b; Rahman et al., 2020; Kpotufe & Martinet, 2018). Despite the fact that implication of the proposed regularization scheme are different. Domain aligning method, such as CORAL (Sun & Saenko, 2016), aligns the correlation matrix from different domains. However, our method (Equation (16) tries to decorrelate each dimension of the representation by minimizing the off-diagonal term of cross-correlation matrix. In our penalty, we do not want to align the covariance between different domains. For example, given batch of data of size B x N, where B is the batch size and N is the feature dimension. Cross-domain covariance tries to deal with the row vector and the correlation matrix is of size N x N. Our covariance penalty deals with column vector, and the correlation matrix is of size B x B. Since usually < N, the computation would be much easier.
covariance well, the motivation and
5.3 RELATION TO CONTRASTIVE-BASEE DOMAIN GENERALIZATION
Our method can also be regarded as a contrastive- learning (Chopra et al., 2005; Caron et al., 2020; Grill et al., 2020; Chen et al., 2020; Zbontar et al., 2021) has been a successful paradigm in self-supervised leaning. Contrastive learning aims at bringing positive pair samples closer together. while moving negative samples further away in a learned embedding space. Essentially, the aim of domain generalization is to extract domain- invariant features, similarly aiming to minimize the distance of features within the same class in the embedding space, while maximizing the distance of features from different classes. Such aim is closely related to the domain generalization. SelfReg (Kim al., 2021) uses only positive data pairs and introduces inter-domain curriculum learning to prevent representation collapse (Grill et al., 2020). (Jeon et al., 2021) uses domain aware supervised contrastive to ensure domain invariance while increasing class discriminability, Compared to previous works, our method instead introduces a much simpler framework to ensure convergence to domain invariant features with theoretical guarantee.
domain generalization problem. Contrastive
CONCLUSION
In this work, we introduce an -theoretinnteooeticaall approach for domain generalization. We cast the task of domain generalization as rate distortion problem and then use information bottleneck penalty to obtain guarantees on the existence of features we want to learn. We link our method, called Twins, with self-supervise learning, which can provide theoretical perspective in the success behind self- supervised learning. We conduct an empirical study on Twins-ERM, and Twins-IRM under various datasets and confirm the consistent improvement of the proposed method over existing baselines. In the future, we intend to further analyze domain generalization in the rate distortion framework and conduct large scale experiments verify our IB formulation in real-world applications.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Kartik Ahuja, Jun Wang. Amit Dhurandhar, Karthikeyan Shanmugam, and Kush R Varshney. Empirical or invariant risk minimization? sample complexity perspective. arXiv preprint arXiv:2010. 16412, 2020.
Kartik Ahuja, Ethan Caballero. Dinghuai Zhang, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of-distribution generalization. arXiv preprint (X1::200600600) 2021.
Michael A Alcom, Qi Li, Zhitao Gong, Chengfei Wang. Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) pose: Neural networks easily fooled by strange poses of familiar objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4845-4854, 2019.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: (907.0v99000099)0 2019.
Benjamin Aubin, Agnieszka Slowik, Martin Arjovsky, Leon Bottou, and David Lopez-Paz. Linear unit-tests for invariance discovery. arXiv preprint arXiv:2102.10867, 2021.
Yogesh Balaji. Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gen- eralization using meta-regularization. Advances in Neural Information Processing Systems, 31: 998-1008, 2018.
Normand J Beaudry and Renato Renner. An intuitive proof of the data processing inequality. arXiv preprint arXiv:1107.072: 107.0740 2011.
Sara Beery, Grant Van Horn, and Pictro Perona. Recognition in terra incognita. In Proceedings of the European conference computer vision (ECCV), pp. 456-473, 2018.
Jan Beirlant, Edward Dudewicz, László Györfi, and Edward Van der Meulen. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6(1):17-39, 1997.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. Advances neural information processing systems, 24:2178- 2186, 2011.
Yochai Blau and Tomer Michaeli. Rethinking lossy compression: The rate-distortion-penr peroooptierreetininnoo tradcoff. In International Conference on Machine Learning, pp. 675-685. PMLR, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 0088220006.0888,, 2020.
Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman. Learning to balance specificity and invariance for in and out of domain generalization. In European Conference on Computer Vision, pp. 301-318. Springer, 2020.
Gal Chechik, Amir Globerson, Naftali Tishby, Yair Weiss, and Peter Dayan. Information bottleneck for gaussian variables. Journal of machine learning research, 6(1), 2005.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), volume pp. 539-546. IEEE, 2005.
Tarin Clanuwat, Mikel Bober- Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:/812.0111 2018.
10

---

Under review as a conference paper at ICLR 2022
L Davisson. Rate distortion theory: A mathematical basis for data compression. IEEE Transactions on Communications, 20(6):1202-1200) 1972.
Aniket Anand Deshmukh, Ankit Bansal, and Akash Rastogi. Domain2vec: Deep domain generaliza- tion. arXiv preprint arXiv: arXiv: 180..0299. 2018.
Jacob Devlin, Ming-Woi Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0488 2018.
Lixin Duan, Ivor Tsang, and Dong Xu. Domain transfer multiple kernel learning. IEEE Transac- tions Pattern Analysis and Machine Intelligence, 34(3):665 479, 2012.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your latent: A new approach to sule-supe learning. arXiv preprint arXiv:2006.07733, 2020.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization, arXiv preprint arXiv:2007.01434, 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- -onggg image translation. In Proceedings of the European conference on computer vision (ECCV), pp. 172-189, 2018.
Seogkyu Jeon, Kibeom Hong, Pilhyeon Lee, Jewook Lee, and Hyeran Byun. Feature stylization and awwaee contrastive learning for domain generalization. CoRR, abs/2200.00856, 2021. URL https: /tta://ariiii arxiv. org/ abs/ 2210 1008. 08596.
Daehee Kim, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. Selfreg: Self-supervised contrastive regularization for domain generalization. arXiv preprint arXiv:2104.0984) 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 142.699 2014.
Andreas Kirsch, Clare Lyle, and Yarin Gal. Unpacking information bottlenecks: Unifying information- theoretic objectives deep learning. arXiv preprint arXiv:2003. 2020.
Jaihyun Koh, Jangho Lee, and Sungroh Yoon. Single-image deblurring with neural networks: A
comparative survey. Computer Vision and Image Understanding, 203:103134, 2021.
Samory Kpotufe and Guillaume Martinet. Marginal singularity, and the benefits of labels in covariate- shift. In Conference On Learning Theory, pp. 1882-1886. PMLR, 2018
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http:://yanp
lecun. www/ exdb/ wwxdd//mmi mmist/..
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang. and Alex C Kot. Domain generalization with adver- sarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5400-5409. 2018b.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624 639, 2018c.
11

---

Under review as a conference paper at ICLR 2022
Yiying Li, Yongxin Yang, Wei Zhou, and Timothy Hospedales. Feature- critic networks for heteroge- neous domain generalization. In International Conference on Machine Learning, pp. 3915-3924. PMLR, 2019.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18. PMLR, 2013.
Li Niu, Wen Li, and Dong Xu. Multi-view domain generalization for visual recognition. In Proceedings of the IEEE international conference on computer vision, pp. 4193-4201, 2015,
Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. arXiv preprint arXiv:2009.00329, 2020.
Mohammad Mahfujur Rahman, Clinton Fookes, Mahsa Baktashmotlagh, and Sridha Sridharan. Correlation- adversarial domain adaptation and generalization. Pattern Recognition, 100: 107124, 2020.
Alexander Robey, George J Pappas, and Hamed Hassani. Model-based domain generalization. arXiv preprint urXiv:2102. 1436, 2021.
Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Nearest neighbor estimates of entropy. American journal of mathematical and management sciences, 23 (3-4):301-321, 2003.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pp. 143-450 Springer, 2016.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (I/W), pp. 1-5. IEEE, 2015.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/00040057. 2000.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Pruceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 5018-5027, 2017.
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. arXiv preprint 1805 2018.
Alfred Wehrl. General properties of entropy. Reviews of Modern Physics, 50(2):221, 1978.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5028-5037, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint 170807770 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03230, 2021.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In European Conference Computer Vision, pp. 561-578. Springer, 2020.
12

---

Under review as a conference paper at ICLR 2022
A PROOF OF THEOREM 3.1
The entropy or the Shannon entropy (Wehrl, 1978) of a discrete random variable X ~ Px with support X' is defined as
H(X) = Px (xx(X = x))-
(19)
ZEX
The differential entropy (Wehrl, 1978) of a continuous random variable Pxx with support X given follows
M(X) = - J
JeEx log (Px(2))
(20)
where dl' x (z) is the Radon-Nikodye derivative of Px w.r.t the Lesbegue measure.
For continuous variables, the differential entropy h(-) is not bounded below, we can define the lower bounded differential entropy (Kirsch et al., 2020) h(X) h(X + =), where is an independent zero-entropy noise E Uniform(O, 1). Since X T E, h(X) 2 h(E) = O. we get that hi-)) bounded below.
Lemma A.1. If X and Y are discrete random variables that are independent with the supports satisfying < [ 20, < |>] oo, where || denotes the number of element in a set, then for
1<
XX(X)+ + H(Y) > AH(X+Y)
(21)
Proof. Define -
H(Z = = Σ Px(x)
Σ
Pxxx (Z = x|xx=)) = log (P = 2/X = =z))
BEX
ZEZ
Σ Px(x)
Σ Pyyx (Y = x|/X log Prixx(Y = =|X =z))
ZEX
ZEZ
Σ (x(x)
Σ Py|x(1 (Y= = x x = log (Prix(Y== = - =|- = =)) (use X LY)
xEX
BEZ
Σ Px(x)
Σ
(Y = x) log =2-
BEX
#EZ
HYY
(22)
Hence,
- -04x))
= HY) (O))XX)))))))-) s HYY)
(23)
X(H( + +Y) H(X)) & XH HYY
(24)
when À
The equality holds if and only if HYY) = 0, which is impossible since 2 2 | < xc.
Lemma A.2. X, and Z are discrete random variables with the supports satisfying S |X| < 2 < oc and 2 < | < co, where |- denotes the number of element in a set. Besides, Y is
independent of X and Z. Then for < 1,
H(X + Y - AM(XYY) -
(25)
Proof. Similar to Equation (22), we would have
(YX+Y)-- - 1(X
(26)
13

---

Under review as a conference paper at ICLR 2022
By the chain rule of conditional mutual information,
(x (X+Y; I(Y; + 2) - 1(Y:
=Y, -0 ( (since Y 1 2) I(Y;++Y) I(Y; + Y) 2 I(Y +Y)
(27)
where the last inequality holds since the conditional mutual information is non-negative. Since À < 1,
I(X + + > ((; X +Y)
(28)
and hence the equality holds iff I(Y; + Y) 0, in other words, Y X Y. If given X +Y Fmax + Vmax. we can infer Y Vmaxx Hence, P(Y Vmax [X + Y mmax + Vmmax 1. However, P(Y = ymax) = 1 as the support of Y has at least two elements, which gives a contradiction. Hence,
T(X ((X+)))))>>>> X((;;+
Lemma A.3. If X and Y are continuous random variables that are independent and have a bounded support, then for X 1,
T(x) + AR(Y) > xi(XYY)
(30)
Proof. Setting ~ Uniform 1) independent of X,Y, we have π(X) = h(X + +.).7(Y)
=
h(Y hex, Y) h(X + Y + c). We have
X(h(X Y +) M(Y + XTXX + and h(X X
(31)
+ +) = ee = T(X + + X)
According to the data processing inquality, (Beaudry & Renner, 2011) 1) and X 1 X + +Y +
tE,
T(X + Y + X) - T(X +e,Y;X) [(X ++;X) where the last equality holds since X + E 1 Y, and we get
(32)
A(h(X + Y + h(Y +))) - 1(X + Y +2; & I(X + xx h(x +
(33)
Rearranging it, we get
T(X) Xi(Y) Xi(X +Y)
(34)
Since A < the equality holds only In the next, we show that it is not possible.
0. In other words, we have X+Y +e 1 X.
The support of X can be divided into the union of intervals. We assume A > 0 such that [xmex A, Fmax] belongs to the rightmost interval of X; and [uaax A, Vmax belongs to the rightmost interval of Y. where mmax and Vmax denotes the maximum of the support of X and Y. Define an
event M Imax + Imax - 8 5 X + Y + S Xmax + Imax + 1. If M occurs, note that e is bounded
by 1, we have
Fx S mmax 8) 0, PY S ymax - 8|.
(35)
8 < A, based on the definition of the interval, we have that
P (X S Tmax - 5) > Py S Vmnx - 8) >
(36)
LY then Py (Y Ymax -8) -8)= === Py (Y S Vmax - 8)M), which is not the case from the above equations (35) and (36).
Lemma A.4. X,Y and Z are continuous random variables that have a bounded support, and Y is independent of X and Z, then for < 1,
7(X + 2IZ)
Xπ x + Y) x xm(X)
(37)
14

---

Under review as a conference paper at ICLR 2022
Proof. Like Lemma A.2, rewrite the inequality
I(X + + X
(38)
Similar to Equation (27), Y is independent of Z we could write T(X Y + I(Y; Y + E). We then use the data processing inequality, we would get
1(X +X + = -
1(X + +
(39)
since À < 1, and similar to the proof of Lemma A.3, ((X +Y + > 0, we would have
IX +
(40)
Proof of Theorem 3.1. The proof of the theorem resembles the proof Theorem 4 in (Ahuja et al., 2021). Consider solution to equation ot
tt X = t. ((Xin: (apu) = Dinv Xinv + T Pspu
X
spu
(41)
= iinv + Papu
A X + Pspu We
and since t achieves the error of g,
I(w Xinv) = 1(2inv Xinv + Pspu Xapu
(42)
In the next we prove 2spu by contradiction. Define o = ([ + Pspu -). S-1 Observe
that we can write Φt X + X Pspu We a) Pspu We 1 क Xe (0+ X
Pine + t spu A Xin and Xinv Wo)
b.1) ++.X*t .x, d Pspu W discrete random variables with finite support of size least two. (discrete case)
b.2) @+ X", Ppp We are continuous bounded random variables. (continuous case)
In the discrete case, from a), b. and Lemma A. it follows that
XH( X²) + H( Wo) > AH( X*)
(43)
Rearranging the terms, we have
H( we) XH(It. (x)) -H X²)
(44)
Since X. = G(Xinv, e), we have H(bi X ( = H (HDppu Wo) and H ( X
0.
Hence, we get
H X Xinv)= - AH W > H( X H( X²).
(45)
and therefore, h X would have lower penalty. In the continuous case, the argument is similar by invoking a), b.2) and Lemma A.3, t can achieve strictly lower penalty than t
Following the proof of the first part of Theorem 4 in (Ahuja et al.. 2021), we can show that $ achieves the same error of q in the training environments. Thus t strictly better solution t1 which contradicts the optimality of कt. Therefore, it follows that 8sp = 0. And hence,
+ T(w inv
X
I( X
(46)
Based on Theorem (Ahuja et al., 2021), solution does not rely on spurious features and satisfies equation (46) for all the points in the support, then under the Assumption 3 such solution solves the domain generalization problem (DG).
15

---

Under review as a conference paper at ICLR 2022
Proof of Theorem 3.2. The major difference here is that we do not condition the unknown invariant feature. Notations are defined similarly as in the proof of Theorem 3.1. Compared with Equation (45), we want to prove that:
H( x1Ix)== AH W*) H( x1xx' H( X²).
(47)
In other words,
H( + W (H spu W > H( X*Xx)
(48)
Since Pspn W is independent of X and 1p+-x= We use Lemma A.2 for the discrete case and Lemma A.4 for the continuous case to prove Equation (48). The rest of the proof is the same as the proof in Theorem 3.1.
B DATASET DESCRIPTION
B.1 LINEAR UNIT TEST
Example 1/1s The dataset in environment E &all is sampled from the following distributions:
Xin
WNi.. (o
ÿe NN... (Wyzr X inv (o))²)
X spu
Nd (Way ge 1),
X + S 3x X sspu),
(49)
2
din Yo,
(dinv + dspu)
where Wys Rdur xdirr, Way E Xo are matrices drawn i.i.d. from the standard normal distribution, Idw. E Rduu is a vector of ones, Ng is dimensional vector from the normal distribution, and S parameter ø is set differently for every environment (i.e., domain). In particular, we set (o== 0.1, (a===))²- = 1.5, and (===))) 2 for the first three environments, In case there are more than three environments, the (a==e)) for j >> 3 is uniformly from Unif(10-2 10). The rotation matrix S set to the identity matrix in Example and a random unitary matrix Example Is.
is a rotation matrix fixed for all environments. The
Example 2/2s Following the notation of the original paper (Aubin et al., 2021), let
Prow = d
Meamel = Neow:
Vanimal = 10 1
(50)
Pgrass Id
Msand -Mgrass;
Pbackground 1.
The dataset in environment e E Eall is sampled from the following distribution:
jd Categorical (pf s" "",(1 s")s", 2p -1- 81-) (1- (1- sn))
(Ndi.. (0, 0.1) + Hcow) Vanimal
if E {1,2], if (3,4)
X
(N (0, 0.10 + Mcamel) Vaniial
(51)
(N 0.1) + (Mgras) Ubackground
if E (1,4). if (2,3).
Xe
spu
(Naep 0.10 + Msand) Vbbrkgronnd
X S. (Xiiv Xepm): Y 1(17 X
where the environment foreground/background probabilities are == = 0.95, == 0.97, pa=ez 0.99 and the cow/camel probabilities are = = 0.3, s=es 0.5, s s = 0.7. For Nenv and j E [3 Reenv - 1), the extra environment variables respectively drawn according to pe=ej ~ Unif(0.9. 1) and ===ej ~ Unif(0.3, 0.7). The rotation matrix S is set to the identity matrix in Example 2 and random unitary matrix in Example 2s.
Example 3/3s The example is meant to present a linear version of the spiral classification problem of Parascandolo al. (2020). Let Pine 0.1 Idw and yr NNa... (0, 1) for all the environments. The dataset in environment e E Eal is sampled from the following distribution:
Ye ~ Bernoulli (
X t S. (Xe Xe spu)
[N (+ 0.1) if 0, WNdi W Miin, 0.1) if y = 1,
N (+/Epar 0.1) if 0,
X
X
(Naap (-MSau 0.1) y == 1,
(52)
16

---

Under review as a conference paper at ICLR 2022
The rotation matrix S is set to the identity matrix in Example 3 and random unitary matrix in Example 3s. In the above dataset, the invariant features are anti-causally related to the label Y
Remark Linear unit test: In the Example 1/ls and Example 3/3s, the invariant features causal and partially informative about the label. The spurious features carry extra information about the label not contained in the invariant features, In the Example 2/2s, the invariant features causal and carry full information about the label.
C GAUSSIAN-FREE ENTROPY ESTIMATION
C.I ESTIMATING ENTROPY BY KNN
Since the feature is in high-dimensional spaces it is challenging to estimate the density of Z, preventing us from directly computing the exact entropy. To remedy this issue, we resort to the particle-based entropy estimator from Singh et al. (2003); Beirlant et al. (1997), which is based on k-Nearest Neighbors (kNN). We introduce this approach in general terminologies. Consider a distribution p with respect to 2 Z, the particle based entropy estimate is given by
1 N
(p(p)
log
log ((k)
log Vol,
(53)
NVol
where & is the digamma function, log k (k) is bias correction term. Vol, is the volume of the hyper-sphere of radius R. |||; - 2% RNN 2|, which is the Euclidean distance between 2; and its k-th nearest neighbor =x. ,KNNN The volumn is given by:
Vol ||
KNN 12 T//2
R T +1)
(54)
where I is the Gamma function, is the dimension of Z. Putting Equation (53) and Equation (54) logether, we have
N
Hp(p)) = n
Σ log N
||
KNN 2/
1l + log N +
(55)
where Ck,n is determined by and k.
C.2 ESTIMATING IB OBJECTIVE 4
Recall that 2 = f( and X -(X,e), we can estimate the IB objective LIB by first sampling {e:} P., 1. S D. and then X ~ p (X*), 1,15 I N, for any fixed i. We can use samples 75 ((X5)) estimate Ex (Z*)) i.e.
Ex. (Z
log 1|
_KNN 1l2 log( Σ No) + Ck.
(56)
Σ N
where n the dimension of 7, 2
D.N; KNN is Z 's k-th nearest neighbor in the full dataset {
j==1j==1-
For the first term, we use the same method but conditioned on the fixed label. In other words, it should be
D
No log [ii
Ex H (2 ~
n
Σ
2 KNNN
1l2 + log N; + Ck.
(57)
where ZNNN is Ziy's 's k-th nearest neighbor in the dataset {Z }
C.3 COMPARISON
We perform our experiments on ColoredMNIST datasets. At each checkpoint, we sample 1024 isntances and set kNN parameter to be 5 to estimate the IB penalty (4) by Equation (56) and (57) at every checkpoints. We plot the trajectory of kNN based penalty in Figure (1), Clearly, our Twins method is able efficiently minimize the true Gaussian entropy.
17

---

Under review as a conference paper at ICLR 2022
ColoredNNIST
CITM
140
baxba panatty
120
100
w mmm
80
00
40
1000
2000
Figure I: kNN based IB penalty (4)
D HYPERPARAMETER SELECTION OVER μ
== 10
10
10
10-1
MNIST
+
80.44 + 2.70
82.83 + 2.73
77.19 = 6.49
72.11 + 4.08
KMNIST
49.48 4.27
52.24
3.94
52.29 3.26
43.02 = 0.68
36.31 + 3,20
FaShionMNIST
54.88 + 1.57
53.87 + 2.41
56.04 # 1.79
52.25 + 2.70
50.96 + 1.61
18


Reviews:
[{"review_id": "LpDO4CpMY2R", "reviewer": "Reviewer_eG9q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well-written and presents the central contributions clearly.\n- The authors list the assumptions in the data-generating process and the algorithm (Section 3) quite clearly.\n- The approach is developed following a consistent and a (mostly) rigorous line of argumentation, with some theoretical guarantees on convergence under suitable assumptions.\n\nWeaknesses:\n- In my opinion, while this work presents some good arguments, its central contributions are not that distinctive from prior work. Minimizing some notion of cross-domain covariance has been studied extensively in both the machine learning and computer vision communities, to varying levels of success (see, e.g., https://openaccess.thecvf.com/content_cvpr_2018/papers/Li_Domain_Generalization_With_CVPR_2018_paper.pdf, https://www.sciencedirect.com/science/article/abs/pii/S003132031930425X for practical examples, and the papers from Samory Kpotufe's group, e.g., https://arxiv.org/abs/1803.01833 that study this in the context of domain adaptation). The central idea is not really inspiring, albeit the connections to rate distortion theory are interesting. Furthermore, it has been shown recently that for real-world problems of domain generalization, it is very hard to beat naive ERM [https://arxiv.org/abs/2007.01434] which makes me wonder whether the assumptions in this paper (and more generally, in a lot of invariant feature research) really hold in practice.\n- This brings me to my second point: The contributions within this paper (both theoretical and the experimental results) do not really further any insight into the domain generalization problem IMHO. The assumptions the authors make are far too simplistic for real-world settings (e.g., zero-main Gaussian feature distributions, which are patently not true for image classification that regularly exhibits long-tailed label distributions), and hence do not provide a complete picture, and can also be at odds with prior work. The experimental results are on handcrafted problems and toy datasets, which again, are too simplistic (and in the case of handcrafted problems, not generalizable to real-world problems). The problem of domain generalization suffers specifically from this issue, where a lot of the proposed work is valid only under certain restrictive assumptions, and usually fail when deployed in practice. It would be nice to see if these assumptions were validated by experiments on data that is not carefully curated and drawn from real-world problems.", "summary_of_the_paper": "This paper studies the problem of domain generalization via invariant feature learning studied from the lens of information theory. The motivation is to use rate distortion theory to obtain a domain-invariant representation. The authors propose an algorithm titled Twins that uses the aforementioned rate distortion theory coupled with a multivariate Gaussian assumption to provide an additional penalty based on cross-correlation, i.e., minimize the distances between pairs of points belonging to different domains but the same label. The authors demonstrate, that under suitable assumptions, this approach will recover the invariant feature representation. The authors construct several hand-crafted problems where indeed this assumption (and the proposed algorithm) work well, and also provide some experiments on MNIST-type data.", "summary_of_the_review": "In summary, I recommend a weak reject. I am quite familiar with the related work in this area and I believe that the authors should put in more effort to identify where their approach fails in the real-world and subsequently see how they can relax assumptions made in the paper by analyzing these failure cases. Domain generalization research has seen a rapid increase in methodological work that propose various lenses to look at the problem, however, it is hard to disambiguate and see the value of these approaches when run in the real-world.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "DXPbkKTv9mB", "reviewer": "Reviewer_DWwT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- It is good that the paper provides the theoretical gurantee of the algorithm for linear binary classiciation.\n\n- After several approximations, the actual contrastive loss formulation in Eq.(14) is adopted. This formulation is closely related to some existing works [A,B] that also employ the constrastive loss for domain generalization. Since they share the similar idea, proper discussion and comparison with them will be necessary.\n\n- The batch construction method and the batch size will directly affect the loss Eq.(14) used for the training. For example, the amount of hard examples included in the contrastive pairs will depend on these choices. It would be good to include the experiemtnal analysis as well as discussion on these factors.\n\n- It is good that the proposed method consistently improves the corresponding baseline methods in two experiments, linear unit tests and mnist-type datasets. However, I have concern that evaluation on more realistic datasets is missing. As the method assumes Gaussian bottleneck which could be a over-simplification for real world problems, I think evaluation on other datasets, such as VLCS, PACS, and OfficeHome, in domainbed would be helpful to show the generalization performance of the method.\n\n- How does the hyperparemter mu affects the accruacy? Analysis of the effect of hyperparameter is missing.\n\n[A] Feature Stylization and Domain-aware Contrastive Learning for Domain Generalization, ACM MM 2021.\n[B] SelfReg: Self-supervised Contrastive Regularization for Domain Generalization, ICCV 2021.", "summary_of_the_paper": "The paper proposes a method for domain generalization. They formulate the problem as a rate-distortion problem and introduce the information bottleneck penalty. Specifically, they realize the objective as a contrastive loss that encourges embeddings across domains to be similar if they share the same label. In the experiments, the method improves the corresponding baselines in several toy datasets.", "summary_of_the_review": "The paper proposes an interesting approach based on the information bottleneck. However, I think the discussion and comparison with related work on more realistic datasets is necessary.\n\n---  \nAfter authors' response  \nI appreciate the authors' feedback. They provided additional results on more realistic datasets (PACS and OfficeHome) and results for different $\\mu$. After reading the feedback as well as other reviews, I would like to keep my score. As also mentioned by other reviewers, the theoretical guarantee of the algorithm is based on quite strong assumptions, which do not likely hold under practical conditions, and I am a bit skeptical about its significance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "774iWqakFT4", "reviewer": "Reviewer_38sD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Weakness: \n\n1) Authors strongly suggest that learning domain invariant features is the only way to solve the domain generalization problem. They prove that  information bottleneck penalty guarantees that domain-invariant features can be learned which in turn guarantees that there is a solution to the domain generalization problem. This may not be completely correct as authors missed out on some of the literature. \n\n2) Weak literature survey. Please cite the papers [1-6] that may be missing from the current paper. \n\n3) Weak experiments section. \n\nStrength: \n\n1) Clear assumption and claims (even through they are strong in some ways, clarity in writing helps understand what authors are trying to convey). \n\n2) Strong justification on why their method could work. \n\n3) Experiments with synthetic data for better understanding. \n\nMain Review:\n\n1) “Domain generalization deals with the difference in the distribution between the training and testing datasets, i.e., the domain shift problem, by extracting domain-invariant features”: This is not correct. Extracting domain-invariant features is just one way to address the domain generalization problem. \n\n2) “The literature on domain generalization aims to learn the features that are invariant across multiple training domains, and to use those features for out-of-distribution generalization.”: Again not correct. Please look at section 2 of Li et al 2019 [1] to get an idea of some non domain invariant learning methods. You could also take a look at [2-5]. These references also should be cited in the current paper.  \n\n3) Assumption 1 and invariant feature map: In “Domain Generalization by Marginal Transfer Learning”, there is no assumption like assumption 1 in the paper. Does this assumption 1 help in anyway to tighten the learning theoretic bound in terms of number of source domains and number of training examples available in the source domains?\n\n4) Assumption 4: “Strictly separable invariant features” This looks like a very strong assumption. What happens if this assumption is not satisfied all the time for all the source domains? \n\n5) The experimental section is not complete as it does not contain most datasets that recent papers use: For example, PACS or VLCS datasets are missing [6]. Can authors get results on one of these datasets at least? \n \n\n\n[1] Li, Yiying, Yongxin Yang, Wei Zhou, and Timothy Hospedales. \"Feature-critic networks for heterogeneous domain generalization.\" In International Conference on Machine Learning, pp. 3915-3924. PMLR, 2019.\n\n[2] Deshmukh, Aniket Anand, Ankit Bansal, and Akash Rastogi. \"Domain2vec: Deep domain generalization.\" arXiv preprint arXiv:1807.02919 (2018).\n\n[3] Niu, Li, Wen Li, and Dong Xu. \"Multi-view domain generalization for visual recognition.\" In Proceedings of the IEEE international conference on computer vision, pp. 4193-4201. 2015.\n\n[4] Blanchard, Gilles, Aniket Anand Deshmukh, Ürün Dogan, Gyemin Lee, and Clayton Scott. \"Domain Generalization by Marginal Transfer Learning.\" arXiv preprint arXiv:1711.07910 (2017) J. Mach. Learn. Res. (JMLR) 22 (2021): 2-1.\n\n[5] Duan, Lixin, Ivor W. Tsang, and Dong Xu. \"Domain transfer multiple kernel learning.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 34, no. 3 (2012): 465-479.\n\n[6] Li, Da, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. \"Deeper, broader and artier domain generalization.\" In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550. 2017.\n", "summary_of_the_paper": "Authors address a problem of domain generalization using  information bottleneck method. Authors assume that learning domain invariant features is a key to solve the domain generalization problem and then prove that  information bottleneck penalty guarantees that domain-invariant features can be learned. ", "summary_of_the_review": "Authors should address the weaknesses of the paper. Improved literature survey and comparing/contrasting with suggested papers will help improve the score. Stronger experiments section can further improve the score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-yRQd-Z5Dp", "reviewer": "Reviewer_t26w", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:** The problem addressed in the paper is very important. It is a good idea to combine the ideas introduced in model based domain generalization [Robey et al.] and the ideas of information bottleneck and invariance [Ahuja et al.].  The experiments show some promise. The paper is easy to read as well. \n\n\n**Weaknesses:** I highlight concerns with the paper below in a pointwise fashion\n\n1. **Access to X to prove theorems makes theorems vacuous** The authors theory assumes access to domain invariant label. If we already have access to X to prove the theorems, the theorems are vaccuous because you can just use X to learn the predictor in any domain and that should work fine. I will describe a data generation process to further my point. \n\n   $X^e \\leftarrow G(X,e)$\n\n   $Y^e \\leftarrow f(X) + N $\n  \n  In the above model the whole difficulty arises because we have access to $X^e$ and not $X$. If we have access to $X$ and have $f$ in the space of functions that the learner searches, then with enough data $f$ would be learned given $X$ as input. Therefore, the theorem in the current version do not make much sense.\n\n2. **Proxy for X is not reliable** The authors for the experiments do not assume $X$ is available and instead rely on a proxy. The proxy is that for an instance $X^e$  the authors use an alternate $X^{e'}$ from a different environment but with the same label. If the authors are borrowing the idea from Robey et al. already why do they not use the $G$ function from [Robey et al.] and use the domain invariant $X$. Take the $G$ model from Robey et al. for each $X^{e}$ find the alternate $X^{e'}$ using their model. Table 1 in [Robey et al.] very clearly illustrates what I am saying. Using the learned model $G$ seems a much more appropriate proxy and not just any point from another domain with the same label.\n\n3. **Regarding experiments** I would encourage the authors to modify their method to incorporate $G$ model in it. However, if the authors insist that their approach is already working and robust, then some more experiments and justification (ablation studies for the same are needed).  Take datasets such as Terra Incognita, Camelyon etc. used in DomainBed and WILDS repository and show that the idea continues to work in more diverse settings. \n\n", "summary_of_the_paper": "In this work, the authors study the problem of domain generalization. Recent works such as invariant risk minimization [Arjovsky et al.] have invigorated a lot of interest in the problem of domain generalization. The paper brings together ideas from rate distortion and information bottleneck principle (recently used in Ahuja et al. for domain generalization) and the work of model based domain generalization [Robey et al.]. \nIn Ahuja et al., the authors combine invariance and information bottleneck to address domain generalization. The representation learned in Ahuja et al. is constrained by the label and information bottleneck constraints. In this work, the authors add some more structure to the generation of the observations X from the latent and say that there is a common instance X that is transformed by the domain transformation model to get the image in the current domain, i.e. X^e = G(X,e). The authors try to recover the common instance X that is shared across domains. \n\n", "summary_of_the_review": "Overall, the paper proposes an interesting approach to combine information bottleneck and model based domain generalization ideas recently used in [Ahuja et al.] and [Robey et al.]. However, there are severe shortcomings in the paper, the theory seems vacuous under assumption of access to $X$ (authors should please correct this). The proxy for $X$ is not reliable and instead $G$ model from Robey et al. should be used to get good results in large scale settings.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Casting domain generalization as a rate-distortion problem and developing an information-theoretic approach to solving it looks like an interesting idea. While the proposed method is technically sound, the assumption made in the proposed method is too strong to hold in real-world applications. Though in the rebuttal the authors provided additional experiments on two benchmark datasets, reviewers' concerns about the strong assumption made in the proposed algorithm still remain. To address this issue, I think besides conducting more extensive experiments, the authors also need to analyze when the assumption does not hold in practice, why the proposed algorithm could still perform well compared with other domain generalization methods.

In summary, this is a borderline paper below the acceptance bar of ICLR.
Decision: Reject