Entropy Penalty: Towards Generalization Beyond the IID Assumption
Authors: Devansh Arpit
Caiming Xiong
Richard Socher
Venue:  (2020)
Keywords: domain shift
information bottleneck
entropy penalty
out of distribution generalization

Abstract: It has been shown that instead of learning actual object features, deep networks tend to exploit non-robust (spurious) discriminative features that are shared between training and test sets. Therefore, while they achieve state of the art performance on such test sets, they achieve poor generalization on out of distribution (OOD) samples where the IID (independent, identical distribution) assumption breaks and the distribution of non-robust features shifts. Through theoretical and empirical analysis, we show that this happens because maximum likelihood training (without appropriate regularization) leads the model to depend on all the correlations (including spurious ones) present between inputs and targets in the dataset. We then show evidence that the information bottleneck (IB) principle can address this problem. To do so, we propose a regularization approach based on IB called Entropy Penalty, that reduces the model's dependence on spurious features-- features corresponding to such spurious correlations. This allows deep networks trained with Entropy Penalty to generalize well even under distribution shift of spurious features. As a controlled test-bed for evaluating our claim, we train deep networks with Entropy Penalty on a colored MNIST (C-MNIST) dataset and show that it is able to generalize well on vanilla MNIST, MNIST-M and SVHN datasets in addition to an OOD version of C-MNIST itself. The baseline regularization methods we compare against fail to generalize on this test-bed.

Under review as a conference paper at ICLR 2020
ENTROPY PENALTY: TOWARDS GENERALIZATION BE- YOND THE IID ASSUMPTION
Anonymous authors Paper under double-blind review
ABSTRACT
It has been shown that instead of learning actual object features, deep networks tend to exploit non-robust (spurious) discriminative features that are shared be- tween training and test sets. Therefore, while they achieve state of the art per- formance on such test sets, they achieve poor generalization on out of distribu- tion (OOD) samples where the IID (independent, identical distribution) assump- tion breaks and the distribution of non-robust features shifts. Through theoretical and empirical analysis, we show that this happens because maximum likelihood training (without appropriate regularization) leads the model to depend on all the correlations (including spurious ones) present between inputs and targets in the dataset. We then show evidence that the information bottleneck (IB) principle address this problem. To do so, we propose a regularization approach based on IB called Entropy Penalty, that reduces the model's '' dependence on spurious features- features corresponding to such spurious correlations. This allows deep networks trained with Entropy Penalty generalize well even under distribution shift of spurious features. As a controlled test-bed for evaluating our claim, we train deep networks with Entropy Penalty on a colored MNIST (C-MNIST) dataset and show that it is able to generalize well on vanilla MNIST, MNIST-M and SVHN datasets in addition to an OOD version of C-MNIST itself. The baseline regularization methods compare against fail to generalize on this test-bed.
INTRODUCTION
now known that deep networks trained on clean training data (without proper regularization) often learn spurious (non-robust) features which are features that can discriminate between classes but do not align with human perception (Jo & Bengio. 2017; Geirhos et al., [018a] [Tipras et al.] 2018 Tlaas al. 2019). An example of non-robust feature is the presence of desert in camel images, which may correlate well with this object class. More realistically, models can learn to exploit the abundance of input-targe target correlations present in datasets, not all of which may be invariant under different environments. Interestingly, such classifiers can achieve good performance on test sets which share the same non-robust features. However, due to this exploitation, these classifiers per- form poorly under distribution shift (Geirhos et af., 200 Sa) Hendrycks & Dietterich) (019)) because it violates the IID assumption which the foundation of existing generalization theory (Bartlett & Mendelson 2002 McAllester] 1999bla). l).
The research community has approached this problem from different directions. In part of domain adaptation literature (Eg. [Ganin & Lempitsky (2014)), the goal is to adapt a model trained on a source domain (often using unlabeled data) so that its performance improves on a target domain that contains the same set of target classes but under a distribution shift. There has also been research on causal discovery (Hoyer et al. 2009; Janzing et al. 20099 [Lope- et al., 2017 Kilbertus al.) 2018) where the problem is formulater Iated as identifying the causal relation between random variables. This framework may potentially then be used to train a model that only depends on the relevant features. However, it is often hard to discover causal structure in realistic settings. Adversarial training (Goodfellow et al. 2014 4| Madry et al. (017)) on the other hand aims to learn models whose predictions are invariant under small perturbations that are humanly imperceptible. Thus adversarial training can be seen as the worst-case distribution shift in the local proximity of the original training distribution.

---

Under review as a conference paper at ICLR 2020
Our goal is different from the aforementioned approaches. We aim to directly learn a classification model using labeled data which is capable of generalizing well under input distribution shift (not constrained to being locally) without making any changes to the model during test time. Thus goal is more aligned with the recently proposed Invariant Risk Minimization (Arjovsky et al. 2019), but imposes less constraints on the data collection process. For detailed discussion on related work, see section 4
Our contributions are as follows:
1. Our theoretical and empirical analysis shows that models trained with maximum likelihood objec- tive without appropriate regularization can, in general, learn to exploit/depend on all the correlations present between inputs and targets in the training set, leading to non-robust representations. While this representation may allow them to yield state-of-the- tion is identical to the training set, they would perform poorly when the distribution of non-robust features shift. This effect is not mitigated by larger training set containing more variations between samples. Thus based on our analysis. should not be surprising that deep networks trained on image datasets show poor performance under input perturbations and (in general) input distribution shifts as discussed in numerous recent papers (Hendrycks & Dietterich 2019 Jo & Bengio 2017 Geirhos a)1
performance on test sets whose distribu-
let al.
2. We provide evidence showing that the information bottleneck (IB) principle Tishby et al. 2000, Tishby & Zaslavsky 2015) is capable of addressing the out of distribution generalization prob- lem. Specifically, our proposal, that we call Entropy Penalty is based on the IB principle and aims at learning representation that throws away maximum possible information about the input dis- tribution while achieving the goal of correctly predicting targets. Intuitively, doing so makes the representation agnostic non-robust features the input, thus allowing the model's predictions to be invariant under a shift of the distribution of such features during test time.
3. We show experimental results using Entropy Penalty in which a deep network trained on colored version of MNIST dataset (see appendix for samples) is able to generalize well on vanilla MNIST, MNIST-M, SVHN and a distribution-shife that most of the baseline methods failed to the extent of achieving performance close to random chance.
version of the colored MNIST dataset itself. We note
2 HOW TO GENERALIZE UNDER DISTRIBUTION SHIFT?
In order to approach solution to our problem, we observe that the out of distribution nature of samples during test time arises because certain aspects of the input change even though the input still corresponds to the same pool of targets as seen during training. An instance of this change would be seeing a camel in the city (during test time) which has dramatically different background features compared to the desert (seen during training). Thus, if a model is trained to depend only on camel features (which defines the class more universally) and ignore other aspects, a shift in the distribution of such aspects will no longer affect the model's decision during test time.
We note that the above intuition is encapsulated by the information bottleneck learning principle (ishbyy et al. 2000 Tishby & Zaslavsky 2015) which minimizes the following objective.
(r) - I(f Y) + (X(
where X and Y represent the input and target (often class label) random variables, fo(.) denotes deterministic model with learnable parameters 0, and / is a syper-parameter While the first term effectively maximizes the training data likelihood, it is the second term that regularizes the model representations to become invariant to non-robust features that are not dominantly present in all samples by minimizing mutual information between input and representation random variables. We now derive Entropy Penalty which is equivalent to the IB regularization for deterministic models. We note that I(),(X), Y) and (X), X) can be written equivalently as follows,
I(fa Y) = H(Y) - (x)))((X)) I (X), X) H(f(()) H((f(())
(3)
where H(.)) denotes entropy and H(---) denotes conditional entropy. Note that H(Y) in Eq. 2 is
fixed and H(Y)fo (X)) is the same as the maximum likelihood loss. Secondly, since fo(-) is a de- terministic function, the conditional entropy Xxfx( is fixed. Thus we only need to minimize
2

---

Under review as a conference paper at ICLR 2020
the entropy of the learned representation (X) in Eq. the minimization in IB objective can be equivalently framed as,
Thus in the case of deterministic fo(.),
(0) = (((Y).((X)) (H)(F(())
(4)
Therefore, we call this general form of regularization Entropy Penalty. Note that while the first term (data likelihood) is easy to optimize, the second term is often intractable for continuous high dimen- sional distributions. The following proposition shows an equivalent form of the above objective.
Proposition! 1 LIB(O) (1- (LL-B))(YYfeeRR + (X)(f(X)) +
where C a positive constant for discrete Y, independent of 0.
The benefit of the above form for (0) instead of Eq. is that it can often be easier to model the conditional Pr(fo (X) compared to the marginal Pr(fe(X)) For instance, if we assume Pr(fo(() Y) is Gaussian for all class labels Y, the entropy of the conditional distribution Pr(Jo(X)) has a closed-form solution given by 0.5 log (2 where of denotes the variance of the class conditional Gaussian distribution of (x) for class
On practical note, when applying entropy penalty to deep networks, we found that it was not effec- tive when applied to the last layer representations. However, applying this penalty to the first hidden layer representation improved performance under distribution shift. While we do not have com- plete explanation for this behavior, we conjecture that this could be because of the data processing inequality for deep networks (Tishby & Zaslavsky 2015) which states,
X) 2 I(h2, X) 2... ....2.4 I(
(5)
where h, denotes the 2th hidden layer representation and I, is the depth of the network. Writing mutual information in terms of entropy and conditional entropy, and taking advantage of the fact that the conditional entropy term fixed for a deterministic conditional, we have that,
(6) Thus entropy is larger for lower layers. Further, minimizing entropy for higher layers does not ensure entropy minimized for lower layers due to the above inequalities. Thus, any excess infor- mation about the input captured by the first layer gets propagated to the higher layers, the effect of which may get amplified under distribution shift if entropy minimization at last layer not done ap- propriately. For all experiments conducted using entropy penalty (EP), use the aforementioned Gaussian assumption on the representation of the first hidden layer and compute its class condi- tional variance in order to minimize the conditional entropy. Specifically, let h(x) represent the first hidden layer representation of an input x (before non-lineari,y), then we implement EP as,
H(h1) 2 H(h2) 2 .... . M(hh)
REP E Ex>DDM/==A)((((X - pxx
where := Ex-D(xxy==))l [h(x)] and D denotes the data distribution. In practice, we re- place expectation with average over mini-batch samples. For CNNs mini-batch has dimensions (B, C, H, W), where we denote B- batch size, C- channels, H- height, W- width. In this case, we reshape this tensor to take the shape (B x 11 x C.) and treat each row as a hidden vector h.
2.1 THRORETICAL ANALYSIS
We now theoretically study the behavior of IB principle two synthetic datasets designed to pro- vide insights into the invariant representation that IB helps in learning, and simultaneously reveals why it should not be surprising that models trained using maximum likelihood (without appropri- ate regularization) perform poorly under input perturbations and distribution shift during test time. Although these analyses are done for linear regression, in each case, we empirically verify these predictions on deep ReLU networks. For our analysis, we use the following objective,
3 [((f(())))))
J(0) = [[(Jo(x) v) + |l111 +
(8)
2Tee
Here the IB regularization H (f)(())lv) is kept in the exponent for the ease of analytical simplicity. Also, setting = yields our baseline case without the IB regularization.
3

---

Under review as a conference paper at ICLR 2020
2.1.1 SYNTHETIC DATASET A
Minimizing the class conditional entropy forces the distribution of neural network representation corresponding to each class to have the minimum amount of information about the input data. Therefore combining this regularization with the traditional classification losses (Eg. cross-entropy) should encourage the neural network to pick features that are dominantly present in class samples and able to discriminate between samples from different classes. To formalize the above intuition, we consider the following synthetic data generating process where the data samples x E IR and labels are sampled from the following distribution,
with probability pi (N(-0,0?) with probability 1 Pi
y ~~ (-1.1}
34
where: i E (1, d). y is drawn with uniform probability, and x = [I' T2. ... x is a data sample. Also, all x; ly are independent of each other. Thus depending on the value of p., a feature D; has a small or large amount of information about the label Specifically, values of p; close to 0.5 do not tell anything about the value of y while values close to 0 and 1 can reliably predict its value. Here we make the assumption that features with p; closer to 0.5 are non-robust features whose distribution may shift during test time, while features with p. closer to and 1 are robust ones. Thus we would ideally want a trained model to be insensitive to non-robust features. The theorem below shows how the model parameters depend on input dimensions for the optimal parameters when training a linear regression model fo(x) == 0 x using the IB objective.
Theorem
Let 0* be the minimizer of (() in Eq. 8 where we have used synthetic dataset A. Then
for a large enough d, 0" = -M 1pp - 1. where M =- Σ + AI + 3(a²1 + Adiag(p @ (1 (1))
FT such that E is a positive definite matrix of Ps & {0, 0.5, 1) for all i.
As implication of the above statement, since M- is full rank matrix, aside from the effects due to Σ (which is data dependent and beyond our control), 0r can in general be non-zero for all input and output correlations. This is especially the case when 3 = 0 (no IB regularization). When using a sufficiently large B, we find that 8 gets reduced for larger values of p; - pt). i.e., when pa closer to 0.5. Thus the IB objective can help suppress dependence of the learned model on non- robust (low correlation) features. Although this analysis is for linear regression, it provides evidence that it should not be surprising that deep networks trained with maximum likelihood objective with- out an appropriate regularization could exhibit a similar behavior. Note that this is not a problem when training and test set are sampled IID from the same distribution, but only becomes one under distribution shift. Also note that since the analysis depends expectation over data distribution, larger training set cannot solve our problem of avoiding dependence on non-robust features.
To verify that the behavior studied above also holds for deep networks, we conduct experiments with both linear and deep models on samples drawn randomly from synthetic dataset A. Details of the experimental setup can be found in appendix B
In figure (left), we plot the parameters 07 vs. P; for the linear regression model. Since the same analysis cannot be done for deep networks, we the perspective that the output-input sensitivity
s", where s = Ex
is equal to 0 for linear regression. So for deep networks, we plot
8f vs. Pt instead as shown in figure (right). In both models, we normalize the sensitivity values so that the maximum value is for the ease of comparison across different 3 values. Both for linear and deep models, we find that the sensitivity profile goes to away from p. = and when applying the IB regularization with larger values of coefficients 19; this effect being more dramatic for deep networks. Thus the IB regularization helps suppress the dependence of model on non-robust (low correlation) features whose distribution may shift during test time, thus allowing its predictions to be invariant to such shifts.
Here additionally note that for a linear regression model, while sensitivity is same as the model parameter 0, it is not merely the first-order sensitivity that gets suppressed for certain input dimen- sions, the output becomes invariant such dimensions altogether. Although this argument does not necessarily apply to deep networks, note that the IB regularization itself enforces a more gen-
This assumption is needed due to technicality.
4

---

Under review as a conference paper at ICLR 2020
1.00
1.00
& = 0 (Linear)
B =
0.75
0.1 (Linear)
0.75
0.1
= 10 (Linear)
B 10
0.50
"ii 0.50
0.25
0.25
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Figure 1: Sensitivity 8 of output fo- (x) with respect to input dimensions xy vs. the probability p; (controlling correlation between input dimension and target) for synthetic dataset (Eq. (9. Left plot shows 02 (same as sensitivity) computed for a trained linear model. Right plot shows sensitivity computed for a trained MLP. IB regularization acts as filter, suppressing the sensitivity of both these models to weak correlation features (p; close to 0.5).
eral condition of finding low entropy representation rather than merely suppressing input sensitivity. Hence the implications of IB could be more general than what our sensitivity analysis shows.
2.1.2 SYNTHETIC DATASET
Using the same intuition that small class conditional entropy induces learned representations to have less uncertainty, given two features that equally differentiate between classes in expectation, the IB objective should pick the one with smaller variance. To formalize this intuition, we consider the following binary classification problem where the data samples Rd and labels y sampled from the following distribution,
[N(y,
with probability p NN,, ko²) with probability
~ f-1,1
(10)
where i {1,2,... d). y is drawn with uniform probability, and [I1, F2. .xa] is data sample. Once again, all 1 y are independent of each other. Thus depending on the value of p; and k, feature x; has small large variance. We would ideally like the model avoid dependence on dimensions with high variance because they are non-robust and a minor shift in their distribution during test time affect the model's decision by large amount. The theorem below shows how the model parameters depend on input dimensions for the optimal parameters when training linear regression model fx(x) 0 using the IB objective.
8 where we have used synthetic dataset Then for a large enough d, 0* M-11, where, M = + 30² diag(p + k(1 - p)), such that Σ is a positive definite matrix.
Theorem Let 0* be the minimizer of (() Eq.
Once again, find that 07 non-zero for all dimensions of the input. Assume without loss of generality that & > 1. Then using a sufficiently large 3 would make the value of 0 approach 0 if p; close In other words, IB regularization forces the model be less sensitive features with high variance. Thus, such a model's prediction will not be affected significantly under a shift of the distribution of high variance features during test time.
To study the extent of similarity of this behavior between linear regression and deep networks, we once again conduct experiments with both these models on a finite number of samples drawn randomly from synthetic dataset B with & = 10 and o² = 0.001. The rest of the details regarding dataset generation and models and optimization are identical what was used section 2.1.1
The sensitivity s vs. plots are shown in figure (left) for linear regression and figure 2 (right) for MLP. the case of linear regression s; = 07_ For both linear regression and MLP, the model's sen- sitivity to all features are high irrespective of p; when trained without the IB regularization (3 = 0) and this is especially more so for the MLP. On the other hand, when training with IB regularization, find that a larger 3 forces the models to be less sensitive to input feature dimensions with higher variance (which correspond to p; = O). The discussion around the generality of the IB regularization beyond sensitivity analysis is same that discussed in section 2.1.1

---

Under review as a conference paper at ICLR 2020
1.0
A= 0
1.0
B=1
0.8
B
0.8
0.6
5. 0.6
0.4
0.4
0.2
0.2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Pi
Figure 2: Sensitivity & of output (x-(x) with respect to input dimensions x, vs. the probability Pi (deciding the choice between feature with variance o² vs. 10o²) for synthetic dataset (Eq. 10.. Left plot shows 0 (same as sensitivity) computed for trained linear model. Right plot shows sensitivity computed for a trained MLP. IB regularization suppresses the sensitivity of both these models to large variance features (p, close to 0).
EXPERIMENTS WITH DATA DISTRIBUTION SHIFT
The experiments below are aimed at investigating: 1. the ability of relevant existing methods to generalize under distribution shift; how well can the proposed method generalize under this shift. Details not mentioned in the main text can be found in appendix
Datasets: We use a colored version of the MNIST dataset (see appendix [A for dataset samples and details) for experiment 1, and MNIST-M (Ganin et al.) 2016), SVHN (Netzer et al.) 2011 lb, MNIST (LeCun Cortes, 2010 in addition to C-MNIST for experiment range and are not normalized. The reason for this is that since we are interested in out of distribution (OOD) classification, the normalization constants of training distribution and OOD may be different, in which case data normalized with different statistics cannot be handled by the same network easily.
All image pixels lie in 0-1
We use ResNet-56 (He et al. 2016b) in all our experiments. We use Adam optimizer (Kingma & Ba 2014) with batch size 128 and weight decay 0.0001 for all experiments unless specified otherwise. We do not use batch normalization in any experiment except for the adaptive batch normalization baseline method. Discussion and experiments around batch normalization can be found in appendix D We do not use any bias parameter in the network because we found led to less overfitting overall. For all configurations specified for proposed method and baseline methods below, the hyper-paramelce learning rate was chosen from {0.0001, 1, 0.001} unless specified otherwise. For entropy penalty, the regularization coefficient is chosen from {0.1, 1, 103.
Other Details:
Baseline methods:
1. Vanilla maximum likelihood (MLE) training: Since there are regularization coefficients in this case, we search over batch sizes from {32, 64, } for each learning rate value.
Variational bottleneck method (VIB, Alemi al. (0166) is existing approximation to the IB objective that uses a non-deterministic network. We therefore investigate its behavior under distribu- tion shift at test time. The regularization coefficient for VIB is chosen from the {0.01, 1,5,
3. Clean logit pairing (CLP): Proposed in Kannan et al. 2018), this method minimizes the e² norm of the difference between the logits of different samples. As shown in proposition (in appendix), minimizing this E2 norm is equivalent to minimizing the entropy of the distribution in logit space under the assumption that this distribution is Gaussian. In contrast entropy penalty minimizes the entropy of the class conditional distribution of the first hidden layer. Due to this similarity, we consider CLP a baseline. The regularization coefficient for CLP chosen from (0.1, 0.5, 1, 10).
4. Projected gradient descent (PGD) based adversarial training (Madry et al.) 2017) has been shown to yield human interpretable features. This makes a good candidate for investigation. For PGD, lint perturbation is used with maximum perturbation from the set {8, 12, 16, 20} and step size of 2, where all these numbers are divided by 255 since the input is normalized to lie in [0, 1] The num- ber of PGD steps is chosen from the set {20, 50). We randomly choose 12 different configurations out of these combinations.
5. Adversarial logit pairing (ALP, Kannan et al. (018 is another approach for adversarial bustness and an alternative to PGD. Since it has the most number of dyyer-parametent we tried a larger number of configurations for this baseline. Specifically,
lnd norm with maximum

---

Under review as a conference paper at ICLR 2020
100
Entropy Penalty
Dataset
Accuracy
PGD
Test Accuracy
80
C-MNIST
96.88
VIB
MNIST
93.75
60
Input Noise
MNIST-M
85.94
AdaBN
SVHN
60.94
20
MLE
0
ALP
2
8 10
12
14
Hyper-parameter Configurations
Table 1: Out of distribution performance on test sets using model trained with Entropy Penalty on C-MNIST dataset.
Figure 3: Performance on the distribution shifted test set of C- MNIST for various methods trained on C-MNIST training set. appendix for samples from C-MNIST dataset.
See figure 5
Validaton Accuracy
100
Test Accuracy
50
50
0
20
40
60
80 Epochs
100
120
140
20
40
60
80 Epochs
100
120 140
Entropy Penalty
CLP
MLE
PGD
Input Noise
AdaBN
ALP
Figure 4: Baseline methods severely overfit color features the C-MNIST training set leading to near 100% accuracy C-MNIST validation set but close chance performance the distribution shifted C-MNIST test set.
perturbation from the set {8, 16, 20} and step size of 2, where all these numbers are divided by 255 since the input is normalized to lie in [o. 1] The number of PGD steps is chosen from the set {20, 50}. The regularization coefficient is chosen from 1, We randomly choose 15 different configurations out of these combinations.
6. Gaussian Input Noise has been shown to have similar effect as that from adversarial training (Ford al., 2019) with even better performance certain cases. We choose Gaussian input noise with standard deviation from the set {0.05, 0.1, 0.2, 0.3).
Adaptive batch normalization (AdaBN, Li et al. (2016) has been proposed as a simple way to achieve domain adaptation in which the running statistics of batch normalization are updated with the statistics of the target domain data. Although this does not fall within our goal of learning a model that does not need any adaption during test time, investigate this method in experiment due to its simplicity. Since there are no regularization coefficients in this case, we search over batch sizes from {32, 64, 128) for each learning rate value.
Experiment 1: In this experiment, we train ResNet-56 on the colored MNIST dataset using the baseline methods and entropy regularization, and test the performance the trained models on the distribution shifted test set of colored MNIST dataset in each case. For each method, we record the best test performance for each hyper-parameter configuration used, and after sorting these numbers across all configurations. we plot them in figure 3 We find that all the baseline methods tend severely overfit the non-robust color features in C-MNIST leading poor performance on the distribution shifted test set of C-MNIST, Figure further confirms this by plotting the validation and test accuracy vs. epoch for all methods for one the hyper-parameter configurations (see appendix for details). Clearly, baseline methods achieve near 100% accuracy C-MNIST validation set but close to chance performance on the distribution shifted C-MNIST test set, showing that these methods have overfitted the color features. Entropy penalty is able to avoid this dependence.
Surprisingly even VIB suffers from this issue. This could be because of improper minimization of the information bottleneck (IB) regularization, which could in turn be due to 1. the reason due to which entropy penalty does not work when applied to higher layers; 2. VIB minimizes an upper bound of the original IB objective. Entropy penalty is able overcome these difficulties.
7

---

Under review as a conference paper at ICLR 2020
Experiments 2:
In this experiment, we hand-pick the model trained with entropy penalty C- MÑIST in experiment 1 above, such that simultaneously performs well on SVHN, MNIST-M and MNIST datasets (see section [ for discussion on this). We used the C-MNIST test set for early stopping. These performances are shown in [ We note that it is non-trivial for a single model to perform well on all datasets with such distribution shifts without any domain adaptation, especially given it is trained on a dataset on which all baseline methods severely overfit to non-robust features.
4 RELATED WORK
Invariant Risk Minimization: The goal of IRM (Arjovsky et al. 2019) is to achieve out of dis- tribution generalization by learning representations such that there exists a predictor (Eg. linear classifier) that is simultaneously optimal across all environments. IRM achieves this goal by learning (stable) features whose correlation with target is invariant across different environments. In other words, if there are multiple features that correlate with label, then IRM aims to learn the feature which has the same degree of correlation with label irrespective of the environment, while ignoring other features. If the representation induced by such stable features, among others, simultaneously also contain the minimum amount of information about the input, then such representations can alternatively be learned using the information bottleneck principle. Thus it boils down to which strategy forms a better inductive bias for handling distribution shift. On a practical note, the main difference between IRM and our proposal is that IRM requires the explicit knowledge of the envi- ronment from which each training data is sampled from. Our approach does not have this restriction. Due to this, we cannot evaluate IRM in our experimental setting.
Adversarial Training: There is an abundance of literature around robust optimization (Wlld 1 1945 Ben-Tal et al. 2009) and adversarial training (Goodfellow et al., 2014] |Madry et al. (017) which study robustness of models to small perturbations around input samples and are often studied us- ing first order methods. Such perturbations can be seen as the worst case distribution shift in the local proximity of the original training distribution. Further, Tsipras et al. (2018) discusses that the representations learned by adversarially trained deep network are more human interpretable. These factors make it a good candidate for investigating its behavior under distribution shift.
Our theoretical analysis has similarities to this line of work, but our goal and conclusions are broader. Specifically, for linear regression, we derive the optimal parameter value analytically under the infor- mation bottleneck objective. Since, the value of parameters is same as the output-input sensitivity- the derivative of this model's output (not loss) with respect to its input, we plot sensitivity in the case of deep networks because parameters do not correspond to input dimensions for deep networks. Nonetheless, this is limitation of our analysis and not of the information bottleneck principle be- cause its objective of minimizing representation entropy is more general than reducing first order sensitivity of the model.
Domain Adaptation: Domain adaptation (Wang & Deng 2018 Patel et al, 2014) addresses the
problem of distribution shift between source and target domain, and has attracted considerabl at- tention in computer vision, NLP and speech communities (Kulis et al. 2011 Blitzer et al. [007]] Hosseini-Asl al. 2018). Some of these methods address this issue by aligning the two distribu-
tions (Jiang& & Zhai 2007 Bruzzone & Marconeini (209)). while others by making use of adversarial
training (Ganin Lempitsky 2014] Ganin et al. (2166) and auxilliary losses (Ghifary et al.) [015]] He et al. (016a). A common characteristic of all these methods is that they require labeled/unlabeled target domain data during the training process. This is not necessary in the information bottleneck approach, which makes it more flexible.
5 DISCUSSION AND CONCLUSION
Based on our analysis, it appears that deep networks are good at achieving state-of-the-art general- ization in common settings because a. they are able to exploit all the correlations present between inputs and targets; and b. the IID assumption holds between training and test sets. However, these attributes also make them perform poorly on distribution shifted test sets. Our analysis provides ev- idence that the information bottleneck (IB) principle can be potential remedy to this problem. We reached this conclusion by introducing entropy penalty- an equivalent form of the IB regularization for deterministic networks, and showing it generalizes well on out of distribution test sets.
8

---

Under review as a conference paper at ICLR 2020
However, note that while entropy penalty itself is general form of regularization, proposed implementation of entropy penalty has certain limitations and it lacks of complete theoretical under- standing. Specifically,
1. The Gaussian distribution assumption of hidden representation is limitation and may not ap- ply to more general datasets other than MNIST, where class samples have multi-modal features. This requires alternate ways of minimizing the entropy of distributions which is currently a hard open problem. Additionally, since entropy penalty works best when there significant gap be- tween correlation/variance of robust and non-robust features (see section 2.T). may not be easy to get good OOD performance when training does not have this property. As evidence, we found this to be the case when training with entropy penalty on SVHN and testing on other datasets (not shown). Two possible solutions this problem could be: a. design more generic algorithms for min- imizing entropy of hidden representation; b. data augmentation techniques that selectively amplify the difference in levels of correlation of robust feature with the target vs. the non-robust ones.
2. Despite our attempt to explain why entropy penalty improves performance under distribution shift when applied the first hidden layer of deep networks, but not higher layers, a more detailed understanding of this phenomenon remains elusive and is left future work.
Disjoint from above discussion, the traditional practice of using a validation set for early stopping and selecting syper-parameters is based on the assumption that rraining/validationnnes sets sam- pled IID from the same distribution (Arlot et al., 2010). However, evaluate out of distribution test sets. This
not clear how to early stop
and select hyper-parre arper-parretter values when the goal because a set of non-robust features can be shared between a training and validation set, and thus high performance such validation does necessarily imply the learned model can generalize to out of distribution test sets. This topic needs further attention.
REFERENCES
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv: 1612.0040 2016.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 1907v:9907.000 000999990.99..... 2019.
Sylvain Arlot, Alain Celisse, et al. A survey of cross-validation procedures for model selection. Statistics surveys, 4:40-79, 2010.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482 2002.
A. Ben- Ben-Tal, L. El Ghaoui, and A.S. Nemirovski. Robust Optimization. Princeton Series in Applied Mathematics. Princeton University Press, October 2009.
John Blitzer, Mark Dredze, and Fernando Pereira. Biographies, bollywood, boom-boxee and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pp. 440-447, 2007.
Lorenzo Bruzzone and Mattia Marconcini. Domain adaptation problems: A dasvm classification
technique and circular validation strategy. IEEE transactions on pattern analysis and machine intelligence, 32(5)777--77,, 2009.
Nic Ford, Justin Gilmer, Nicolas Carlini, and Dogus Cubuk. Adversarial examples are a natural consequence of test error in noise. arXiv preprint arXiv:1901.100 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. arXiv
preprint arXiv:1409 140..449..9 2014.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-ade anvasiaadversarial training of neural net- works. The Journal of Machine Learning Research, 17(1):009-2000 2030, 2016.

---

Under review as a conference paper at ICLR 2020
Robert Geirhos, Patricia Rubisch, Claudio Michaelis. Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint urXiv:/811.123311 2018a.
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks, In Advances in Neural Informa- tion Processing Systems, pp. 7538-7550 7550, 2018b.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generaliza- tion for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pp. 2551-2559, 2015.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint
2014.
Di He, Yingee Xia, Tao Qin, Liwei Wang. Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. Dual learning for machine translation. In Advances in Neural Information Processing Systems, pp- 820-828, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016b.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint vrXiv:1903, /2261, 2019.
Ehsan Hosseini-Asl, Yingbo Zhou, Caiming Xiong, and Richard Socher. A multi-
discriminator cyclegan for unsupervised non-parallel speech domain adaptation. arXiv preprint arXiv:1804.00522. 2018.
Patrik O Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlin- ear causal discovery with additive noise models. In Advances in neural information processing systems, pp. 689-696, 2009.
Andrew lyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. arXiv preprint urXiv: peee:09051277 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Dominik Janzing, Patrik O Hoyer, and Bernhard Schölkopf. Telling cause from effect based on high-dimensional observations. arXiv preprint arXiv:0909.4338. 2009.
Jing Jiang and ChengXiang Zhai. Instance weighting for domain adaptation in nlp. In Proceedings of the 45th annual meeting of the association of computational linguistics, pp. 264-271, 2007. 2007.
Jason Jo and Yoshua Bengio. Measuring the tendency of cnns to learn surface statistical regularities. arXiv preprint arXiv:1711.11550 2017.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv: (00::1803.000 06373, 2018
Niki Kilbertus, Giambattista Parascandolo, and Bernhard Schölkopf. Generalization in anti-causal learning. arXiv preprint 1rXiv:1812.00524, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
1rXiv:1412.6980 2014.
Diederik P Kingma and Max Welling. arXiw:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Brian Kulis, Kate Saenko, and Trevor Darrell. What you saw is not what you get: Domain adaptation using asymmetric kernel transforms. In CVPR 20/ I, pp. 1785-1792. IEFE, 2011.
10

---

Under review as a conference paper at ICLR 2020
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. http://////
lecun. com/ wwtp/edo///o.ecceeoooooee exdb/mnist/
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou, Revisiting batch normaliza- tion for practical domain adaptation. arXiv preprint arXiv:1603.00 04779, 2016.
David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou. Dis- covering causal signals in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6979-6987,, 2017.
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0608. 2017.
David A McAllester. Pac-bayesian model averaging. In COLT, volume 99, pp. 164-170. Citeseer, 1999a.
David A McAllester. Some pac-bayesian theorems. Machine Learning, 37(3):355-363, 1999b.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Visual domain adaptation: An overview of recent advances. IEEE Signal Processing Magazine, 2014.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/00040057, 2000.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint 1805 2018.
Abraham Wald. Statistical decision functions which minimize the maximum risk. Annals of Math- ematics, pp. 365-280, 1945.
Mei Wang and Weihong Deng. Deep visual domain adaptation: survey. Neurocomputing. 312: 135-153, 2018.

---

Under review as a conference paper at ICLR 2020
0
1
/
2
?
2 3
3
3
4
H
4
4 5
4
5
5
S
5
S
s
5
5
5
5
6
6
6
6
6
6
co
6
6
7 7
7
7
7
7
8
8
8
8
8 7
6
8
g
8
9 9
Figure 5: Color MNIST training set (left) and out of distribution test set (right). Each class in training set has two background colors and two foreground colors that are unique only that class. Each test image has foreground and background color that is randomly picked out of 10 colors that are chosen independently of the training set.
4
1
3
5
8
9
3
9
2
6
2
2/6
8
1
10
8
4
5
Figure 6: Random samples from MNIST (top). SVHN (middle) and MNIST-M (bottom) datasets are shown get a visual sense of the hardness of the of distribution task.
APPENDIX
A DATASETS
Randomly drawn training and test samples from the C-MNIST dataset generated as described above are shown in figure The colored MNIST dataset (C-MNIST), which used in experiments 1 and 2, uses all the 60,000 training image in the MNIST dataset generate the C-MNIST dataset, where we randomly do 0.9-0.1 split to get the training set and validation set respectively. Similarly, we use all the 10,000 test images in MNIST to generate the C-MNIST test set. We vary both the foreground and background colors to generate our C-MNIST dataset. The reason why we vary colors of both foreground and background that we want the trained model to avoid overfitting any color bias. A single color in foreground or background would constitute a low variance feature, which as we study in section 2. 1.2 leads models to prioritize learning it for both training with vanilla MLE well as MLE with entropy penalty.
Colored MNIST Dataset:
The C-NNIST training/validation set is generated from its MNIST counter-par conter--pat& as follows:
1. For each class, randomly assign two colors (RGB value) for foreground and two colors (RGB value) for background.
2. Binarize each image (pixels in 0-255 range) around threshold 150 that pixel values are either
0 or and replicate the channel in each image to have a three channel image.
3. For each image in class, randomly pick one of the two foreground colors assigned to that class and replace all foreground pixel with that color. Similarly replace background pixels for all images.
4. Add zero mean Gaussian noise with small standard deviation (0.04 used in experiments) to all images.
To generate the test set, in step one, randomly assign foreground and background color each image irrespective of the class, and the colors for validation set are chosen independently of the training set.

---

Under review as a conference paper at ICLR 2020
100
AdaBN (LR 0.001)
Test Accuracy
80
BN (LR 0.001)
60
AdaBN (LR 0.0001)
BN (LR 0.0001)
40
No BN (LR 0.001)
20
No BN (LR 0.0001)
0
15
30
45
60
75
90
Epochs
Figure 7: Batch Normalization is unstable without AdaBN when using Entropy Penalty (experiment on C-MNIST dataset).
Other Datasets: During experiments with MNIST, the single channel each image was replicated to form three channel image. For SVHN, we resized the image to have 28 28 hieght and width.
B EXPERIMENTAL DETAILS
Experiment in Section 2.1.1 The input data is in 500 dimensions and set the value of for each i to be uniformly from [0, 1 and fix henceforth. We then randomly sample 15000 input-target pairs from this dataset with 0² 0.0001. We train linear regression model and a hidden layer perceptron (MLP) of width 200 with ReLU activation for 1000 iterations using Adam optimizer with learning rate 0.001 and weight decay 0.00001.
Variational bottleneck method (VIB): As an implementation detail of VIB, we flatten the output of the last convolution layer of Res and separately pass it through two linear layer of width 256, one that outputs a vector that we regard as mean μ,, and the other that we regard as log-variance (similar how is done in variational anto-enccders (Kingma Welling 2013)). We then combine their outputs as μ + erp(0.5vv O where is sampled from standard Gaussian distribution of the same dimension as v. This output then passed through linear layer that transforms into vector of dimension same as the number of classes. These implementation details are similar to those in the original VIB paper.
Further, gradually ramp up the regularization coefficient 3 from 0.0001 to its final value by doubling the current value at the end of every epoch until the final value is reached. This popular prctice when minimizing the KL divergence term between posterior and prior which helps optimization.
C OVERFITTING COLOR FEATURES
The validation set of C-MNIST is a held out set that follows the same distribution as the training but the two are mutually exclusive. On the other hand, the test set C-MNIST has different foreground and background colors in all images that are sampled independently of the training set as explained in section A and shown in figure 5 The hyper-parameter configuration chosen (from among the ones used in experiment in section 3) for each method is based on the condition that training accuracy converged to 100% the end of the training process. This ensures a fair compar- ison of validation and test accuracy between different methods.
D EFFECTS OF BATCH NORMALIZATION ENTROPY PENALTY
In this section study the effect of batch normalization (BN, loffe & Szegedy (2015) when using in conjunction with entropy penalty. We train ResNet-56 on C-MNIST dataset identical to the settings in experiment 1 in main text, with the exception that BN. While doing so,
2

---

Under review as a conference paper at ICLR 2020
record the accuracy of the model on the C-MNIST test set at every epoch. In addition, we also run experiments where under the same training settings, during evaluation at each epoch, we use AdaBN (Li et al,, 2016). These values are plotted in figure 7 We have also plotted runs using entropy penalty without any batch normalization for reference. In all cases, we use a learning rates (LR) of 0.001 and 0.0001. The plots show that without AdaBN, test accuracy is very unstable on the distribution shifted test set. This seems to be a sidc-effect of using BN which can be fixed by adaptive the running statistics of BN (used during evaluation) with the test set statistics. However, this requires domain knowledge (i.e., samples) of the test set, which is not preferable for our goal.
E PROOFS
Proposition 2 (0) (1-33(1((( + (H(J)(X) +C
where C is a positive constant for discrete Y, independent of o. Proof: We note that,
/ log
H())
(1I)
= =- --/ / (x( Y) log Pr(f(((()))))
(12)
Σ / Pr(fo(X)) Y) (10g (1ogPr(fo(X))))) + log Pr(fo(X))
Pr(fo(X)\Y) )
(13)
Σ / Pr(fo(X), (log log (r((()/(X))))) + log Pr(Y))
(14)
(X)) H(Y (((())(X)) + H(Y)
(15)
where the fourth equality due to Bayes rule. Since Y discrete, H(Y) is a positive constant. Finally, substituting the above expression in the definition of CIB(O), yields.
(I) H.(YY ((HL)))))))))) H(Yl.f( H(Y))
(16)
= (1 -1-33)8
(17)
which proves the claim.
Lemma 1
B|x yy = -B[xlly - -] 2p;
(18)
E| by = 1]=1El=?? -1 o²
(19)
Proof:
Given the distribution of x, we can write each element x, as,
34 ben + b.)n,
(20)
where b; is sampled from the Bernoilli distribution with probability pt, n, N(y, o²), and ñz ~
N(-y,o²), ).). Thus,
y,
E[z,ly= = 1] p; (1-p:) = =2p; -
(21)
E[z y = 1] -p: + (1 p:) = - 2ps
(22)
Next,
Blo 1] (1
b. + +b+bb1 bb)nn,ū.ly
(23)
We know from the properties of Bernoulli and Gaussian distribution that,
Ebb31 var (b.) ).)= + E[b,]² p. - p.) + _/ pt
(24)
E Einfly = var (n.ly = 1) 1 02 +1
(25)
3

---

Under review as a conference paper at ICLR 2020
We similarty get,
Bin 11 0² +1 Bin y 1] 0² +1 E Ein y - = 0² +1 +1
(26)
(27)
(28)
Therefore, using the independence property between random variables,
= - +1 - 22 -E0 = 1]
+ 2([66, -
= 1|
(29)
1 +
(30)
We similarty have,
Bslell -1] =1
(31)
Theorem Let 6t be the minimizer of J(0) in Eq. 8 where we have used synthetic dataset A. Then for large enough d, 0" is given by.
8 1
(32)
where,
M: E E B(021 - 4diag(p @ - (1-p)))
(33)
such that Σ a positive definite matrix
P: & {0, 0.5, 1} for all
Proof:
We start by noting that,
H()(())u)= = Σ /
(34)
8E1-1.1)
0.5 / 10 - = 111)
log (f(e((x)
-1)
(35)
Identifying that,
Hso0x]l==1:=- / Pr(fo(x)ly 1 Pog =
(36)
ffariri====fppaeee Pr(fe(x)ly = -1) [(Pr(fe(x) Pr(fe(x|y = -1)
Hjst))lp=-1
(37)
the entropy of the class conditional distributions Pr(fe(x)|M 1) and Pr(fo(x)|y -1)
respectively, we have that,
H(Jo(x)lu) = 0.5 (H + H (10())/==11)
(38)
We note that,
fo(x))(y = 0,2i1y= =1
(39)
1) (1) (1 1))
(40)
For a large enough dimensionality d of central limit theorem (CLT) applies to fo(x) = 1) and converges to a Gaussian distribution. Thus the entropy of this distribution given by
Thii assumption needed due technicality.
4

---

Under review as a conference paper at ICLR 2020
0.5 log(2(ess} where si is the variance of fo(x)ly = 1. A similar argument applies fo(x)ly=
-1, in which case we define s²1 to be its variance. Thus,
of var
0, /y/ 1)
(41)
d
var (0,t; = 1)
(42)
d
0 var (a:lx= = (x:ly=11 =1
(43)
where the second equality holds because 2; 's are independent of one another. Thus using lemma
7
s = Σ ff(11 + - -(pp- 10k-11 1)))
(44)
Σ (0² + +4pi(1 -p:))
(45)
We similarly get,
22 E09l0²
06(02 + +4p:(1 pt))
(46)
Since of and s²1 are equal, we denote 8² = of = 3²11 Thus,
Ho(00ly=1 = 0.5 log (2
(47)
H 0.5 0.5 logeees
(48)
and,
H(Jo(x)lv) 0.5(0.5logg2 (og(eres>)) + 0.5 0.5 log )
(49)
(50)
Therefore, our objective becomes,
arg min [(fo()) y) + 210122 +3s
(51)
0
arg min 0T E[xxT]0
28 E[yy] + 2110112 + 3 Σ 00 + 4p.(1 (p))
(52)
Define M as,
: E + XI 3(a21 - 4diag 2 (1 (1))))
(53)
where Σ = E[xxf], we can re-write our objective as,
arg min 0T M0 - 20 E[xy]
(54)
whose solution is given by,
8" M M- -
(55)
Using lemma 7 we get,
Elayy] E[r: Pr 1) E|a.ly Pryy
(56)
O.5(EE = 1] -E(2/y/ 1])
(57)
0.5(4p; -2) 2pp
(58)
Plugging this value in Eq. 55 yields 0".
5

---

Under review as a conference paper at ICLR 2020
Now we prove that Σ is full rank. Note that it is positive semi-defiitte since it is scatter matrix. Next, due to conditional independence, for + J.
E [Bgra = B| y = 1] Pr(y = 1) + E -1
(59)
Blayly 1| Pr(y +
(60)
for't =j j.
E[aî] = E[-ily
= 1) Blully Pryy
(61)
Using lemmal
we get,
o2 Eg (11 2p:) 2p))
i otherwise
(62)
To prove that E is positive definite (and hence full rank), we need to prove that no two columns are parallel. To show this, consider any two indices and j such that i + j. We show that there exists no * 0 such that the columns Σn aLig. We prove this by contradiction. Suppose En = a'i
then oزj EnZo, Substituting values from Eq. 62
Ege
(1 + 0²)²
aEii = (1 2p:)( -2p))
(63)
Thus aZzz > 1. However, Eij (1 21 2pz) < Thus there is non-zero o for which
E, = aLjj. Hence E must be full rank and hence positive definite. Thus we have proved the claim. O
Lemma 2
E E[z,ly -
EErr,y 11 =
(64)
Biaily 1] Blaõly -1 1 jpi kz pi))
(65)
Proof: Given the distribution of x, we can write each element v; as,
34 ber + (1 -1-00 b.)n
(66)
where by is sampled from the Bernoilli distribution with probability pt. n, N(y,o2), and n.
N(y, k, Thus,
E[ra ly = 1] = + (1 (1-10 pp)) =
(67)
E E[rily =11]= - - p:) -
(68)
Next,
Blo yy = =] (1 nm + 2b - b.)m,,ūzly = 1]
(69)
We know from the properties of Bernoulli and Gaussian distribution that,
E B = var (b:) + E[b,]] = ps p.) + + Pi
(70)
E E/n = = var =)=
(71)
We similarly get,
L3 En - = 0² +1
(72)
E Ei 1] = ko +1
(73)
E n by -1] ko +1
(74)
Therefore, using the independence property between random variables,
Eiaîly= = 1 2E =]
11 + +(((B() -
(75)
= -p - (1 (1- xk02)
(76)
We similarly have, Rearranging these terms yields the claim.
kk²)
(77)
0

---

Under review as a conference paper at ICLR 2020
Theorem 4 Let 6* be the minimizer of J(0) in Eq. for a large enough d, 0 is given by.
8
where we have used synthetic dataset B. Then
0" = 1
(78)
where,
M -+=++A+ Bo diag(p + k(1 p))
(79)
such that Σ is a positive definite matrix.
Proof: Similar theorem 1
we have that,
H(fo(x)lu) = 0.5 (Hool + H1o(
(80)
and,
fo(x))(y =) fo(x)((=1)= 1) - ()) (i) 1)
(81)
For large enough dimensionality d of x. central limit theorem (CLT) applies fo(x)](y 1) and converges to a Gaussian distribution. Thus the entropy of this distribution given by 0.5 log (og((()))) where sf is the variance of fe(x)ly = 1. A similar argument applies fo(x)ly -1, in which case we define s²1 to be its variance. Thus,
d Σ
$7
pvar(x:ly =
(82)
Thus using lemmal
3
Σ
(83)
We similarly get,
Σ 00 + 11, p:)))
(84)
Since oî and s²1 equal, we denote s² = ot 5² Thus,
H1(0))== 0.5 log (2Tes
(85)
H((())/==1 = 0.5 log (2Tes²
(86)
and,
H(fo(x)lv) = lo500.5log(2Tess + 0.5 (og(eee)))
(87)
0.5 logooo2esss (2Tes²)
(88)
Therefore, our objective becomes.
arg min [[(fo(x) y) + 2l10ak + Bs²
(89)
= arg min 8 E[xx" - 28 28' E|xy] + x|1l|2 + 30² 0p + tb,(1 - p.)
(90)
Define M as,
: E Bod + k(1 - -p))
(91)
where Σ = B[xx=],, we can re-write our objective as,
arg mig 0t MO 20 E[xy]
(92)
9
whose solution given by,
0" = M -
(93)
7

---

Under review as a conference paper at ICLR 2020
Using lemma 2 get,
D Elayy|
= B[ (. = - -1]
1] Pr(y = = - -1)
(94)
(95)
(96)
Plugging this value in Eq. 93 yields 0"
Now we prove that Σ is full rank. Note that it is positive semi-defintt since it is scatter matrix. Next, due to conditional independence, for + j.
I M = Baaan = Pr(y = =) -1) -1)
(97)
Bi
1| -|
-
(98)
and i =
Bia,] Bi-î
Pr(y = Bla = Pr(y
(99)
Using lemma 2 we get,
Σi = {1
[1+ + 14 (p)) [1
if otherwise
(100)
To prove that E is positive definite (and hence full rank), we need to prove that no two columns are parallel. To show this, consider any two indices and j such that i + j. We show that there exists no * O such that the columns Σ, a. We prove this by contradiction. Suppose Eii a.ii. then aIك EnEs,, Substituting values from Eq. 100 Ezs
Diiz = (1 0² 22 + k(1 P1)))( o + + -P)))
(101)
Thus aEgz > However, Dij 1. Thus there no non-zero a for which E; oLD Hence E must be full rank and hence positive definite. Thus we have proved the claim.
Proposition 3 If Pr(ff follows a Gaussian distribution, then, H(fo (X)) = 0.5 (og(TelExx,... XxxDDX-DDDXXX Xx (X2))
(102)
where X1 and X2 IID) samples from the data distribution D(X).
Proof: We note that,
H(Jo(X)) =
log log
(103)
- Pr(Jo(())) log Pr(fooXX
(104)
-0.55 lo..llog22000"
(105)
where o² is the variance of the Gaussian distribution Pr(fo(X)).
Let = Ex,~Do Ex, Then note that Exx~D(X)(/ol (X2) as well since X, and X, are
IID samples. Therefore,
Xxxxxxx (o())))
(106)
D(X μ4) (Fo(X2) 11))
(107)
[xxxxxxx(((((x.) - - [Xx - =1))
-2Ex, (BXx)xxxD(0)((((((X() - (x)(F(X2)) - -w]]
(108)
= (Fx -x)
(109)
= 20"
(110)
Therefore substituting Ex, (Xx_XXXXX(((0000 ffo (X in Eq. 105 yields the claim. D
8


Reviews:
[{"review_id": "SklQj-86tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the author(s) presented a regularization scheme that intends to suppress the identification of spurious features when learning deep representations. Their construction was inspired by the information bottleneck framework. By making Gaussian assumptions on the form of label conditioned feature distribution, the entropy penalty can be efficiently computed in the form of L2 loss, which is easy to implement. My major concerns for this submission are its clarity, novelty, and theoretical depth. The arguments provided are not very convincing and reported experimental results are based on toy-scale datasets. I recommend rejection for this submission, with more detailed comments attached below. \n\nStrength\n+ The author(s) are trying to resolve the issue of learning spurious discriminant features for predictive models, which is a trendy topic with a potential impact on the field. \n+ There are some interesting discussions in the related work section. \n\nWeakness\n- The presentation needs to be much improved. In its current form, the lack of clarity leads to serious confusion. Examples include but not limited to the following:\n\t- \"violates the IID assumption which is the foundation of existing generalization theory\". Not sure what this IID assumption means, please briefly/intuitively describe these classical generalization theories. \n\t- \"all the correlations btw inputs and targets.\"\n\t- \"throws away maximum possible information about the input distribution\"\n- The author(s) have made a strong statement, quote \"it is the second term that regularizes the model representations to become invariant to non-robust features\"\n- Eqn (1) and Eqn (3) is equivalent, what's the point??? There is no novelty here. \n- Prop 1. Modeling the conditional entropy H(f_{\\theta}(X)|Y) nonparametrically is not any easier than modeling the marginal entropy H(f_{\\theta}(X)). The assumption of a parametric form of f_{\\theta}(X) given Y is very strong and needs to be justified (at least experimentally). Although the author(s) are honest about this limitation in the discussion. \n- The concept of distribution shift is not formally introduced in the manuscript. \n- Eqn (7) implicitly makes a strong prior assumption that the feature distribution condition on the label is isotropic Gaussian. This reminds me of Linear Discriminant Analysis (LDA), which followed from a similar heuristic, and might partly explain the empirical success of this practice (the model is forced to be LDA like, which combats the overfitting via appealing to simpler models). However, I have not found any discussion related to this, which evidence that the author(s) might lack a proper understanding of classical treatments.  \n- Theoretical analyses of synthetic examples do not lend strong support to this paper. \n\nQuestions\n# What is the fundamental difference btw the proposed work differs and domain adaption?\n\nMinors:\n% Conditional entropy H(f_{\\theta}(X)|X) is zero. \n% I do not see the point of referencing adversarial robustness literature. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeSUXxRFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed Entropy Penalty (EP) training, based on Information Bottleneck (IB) to make the trained model generalize beyond the IID assumption that is satisfied by usual training and testing datasets.\n\nFirst, a loss function derived from IB is provided Prop. 1. Then they argue that minimizing this loss over lower layers is better Eq. (5) and (6). They further assume the hidden layer is Gaussian, and use squared l2 loss as entropy penalty.\n\nThey theoretically analyze a different loss function Eq. (7) under two simple cases, where optimal solutions have closed forms, and find that the optimal solutions contain smaller weights for non-discriminative features (p_i there).\n\nExperiments on coloured versions of MNIST are conducted to show that the proposed Entropy Penalty achieves better results than several baselines.\n\n1. There is no definition of a robust or non-robust feature, but just a vague description by a simple example (camel appearance). I suggest before presenting the method, providing examples like p_i the synthetic cases, to give a sense of what kind of features are (non-)robust.\n\n2. The Gaussian assumption of the hidden layer is quite restricted as mentioned. And why only apply EP on the first layer rather than all the hidden layers?\n\n3. The objective in analysis Eq. (8) looks different from Eq.(7). Why not using the original objective?\n\n4. The synthetic examples are not convincing enough. For example, what are the solutions of other methods, and how can we see the EP solution is better than others for these examples. More comparisons are needed to give a sense of the advantage of EP.\n\n5. Why is randomly assigning colours considered as a distribution shift? It is not clear to connect what the synthetic examples try to deliver and the coloured version of MNIST here. \n\nThis paper shows that EP based on IB, together with the Gaussian assumption of the hidden layer can learn robust features as shown in synthetic case analyses and coloured MNIST experiments, comparing with several baselines. However, there are several gaps,\n\n1) what exactly is a non-robust feature, in synthetic examples, these are p_i and k, however, in experiments, it seems to colour. There is no definition and thus it fails to show what exact kinds of features the proposed EP actually can capture (p_i, k, colour, or something more general). In this sense, I have the impression that the presentation is not very clear (EP can learn something, but what it is?).\n\n2) The claim is that EP learns robust features for deep learning methods, but both the analyses and experiments are not enough to show that. First, several restricted assumptions are made as mentioned, learn the first layer only, Gaussian assumption, and the examples are too simple, and lack of calculation for other methods and comparisons. Second, just coloured version of MNIST is not convincing to show that EP captures many/most robust features, even though the proposed EP significantly outperforms the other baselines here because of there probably (must) have many different kinds of other features. More experiments are needed to support the efficacity of EP.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJghkMLaYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to combat domain shift via the information bottleneck principle, where the representations are encouraged to learn features that have high mutual information with the labels while having low mutual information with the original data. The paper proposes a specific approach to enforcing this information bottleneck, called \"entropy penalty\", which tries to minimize the L2 distance between the representation at the first layer and the mean representation of the corresponding class. \n\nThe paper proceeds to demonstrate theoretically that some variant of the proposed entropy regularization works for synthetic, binary cases in the sense that it decreases the effect of features that are not highly correlated with the label.\n\nThe paper finally demonstrates superior performance on synthetic experiments and classifying digits in an out-of-distribution setting where the OOD distribution is changing foreground and background color.\n\nMy greatest concern of the proposed approach is the generality of the method. While the information bottleneck is a very general principle, the way it is implemented in this paper is very specific, which only works for the first hidden layer representation of an input x (even before non-linearity). Combining this with the experiment in which the domain shift is changing colors of in the images (which applies linear transforms to background and foreground pixels), it is hard for me to believe that the same procedure is not tailored for this specific type of domain shift (although this is still an interesting type of domain shift that is important in sim-to-real applications in robotics). \n\nIn fact, based on the information bottleneck principle, one could essentially learn that the color is also a feature that also has high MI with label and low MI with inputs, thus the top-left pixel would be indicative of the label. If I use this as the feature for entropy penalty (assuming that there is 1 color per class), this gives a R_{EP} loss of zero. I wonder if this is the reason why the experiments specifically asked for two foreground and two background colors for each class.\n\nBased on there concerns, it might seem more helpful to consider other types of \"unknown\" domain shifts, such as CIFAR vs. CINIC, in which the method proposed might introduce fewer inductive biases than in the case of being invariant to color. \n\nMinor comments:\n\t- It would seem like L1 regularization would achieve a similar effect to what is shown in the theory here? It would be interesting to see how sensitivity changes with some existing regularization methods, because they can be easily implemented.\n\t- Entropy is smaller for lower layers?\n\t- Minimizing entropy for higher layers at least minimizes an upper bound to the entropy of lower layers (if we consider discrete RVs).\n\t- Eq 6 is false for continuous RV and differential entropy. One could simply have h_2 = 2 * h_1 to get larger entropy. Eq 5 is true from data processing but the conditional entropy H(h_l | x) is not fixed.\n\t- It seems that from the discussion about entropy / MI of different layers does not justify the EP approach for the first layer. Even if the final layer retains all the information, you can still technically apply EP to remove redundant information? You can use leaky relu activation to make sure information is not lost in activation functions.\n\t- Typo for definition of \\mu_k\n\t- The first layer is a convolution layer, so maybe it could be helpful to visualize the learned features with or without EP?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an entropy penalty related to information bottleneck to deep neural network regression problems. The reviewers had a number of questions and concerns about the paper, which the authors did not address. In light of this, the reviewers agree that the paper is not yet ready for publication. Please carefully read and address the reviewer's concerns in future iterations of this paper.
Decision: Reject