Information Retention via Learning Supplemental Features
Authors: Zhipeng Xie
Yahe Li
Venue: ICLR 2024 spotlight (2024)
Keywords: Information Retention
Few-shot Learning
Deep Neural Network
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: The information bottleneck principle provides an information-theoretic method for learning a good representation as a trade-off between conciseness and predictive ability, which can reduce information redundancy, eliminate irrelevant and superfluous features, and thus enhance the in-domain generalizability. However, in low-resource or out-of-domain scenarios where the assumption of i.i.d does not necessarily hold true, superfluous (or redundant) relevant features may be supplemental to the mainline features of the model, and be beneficial in making prediction for test dataset with distribution shift. Therefore, instead of squeezing the input information by information bottleneck, we propose to keep as much relevant information as possible in use for making predictions. A three-stage supervised learning framework is designed and implemented to jointly learn the mainline and supplemental features, relieving supplemental features from the suppression of mainline features. Extensive experiments have shown that the learned representations of our method have good in-domain and out-of-domain generalization abilities, especially in low-resource cases.

Published as a conference paper at ICLR 2024
INFORMATION RETENTION VIA LEARNING SUPPLE- MENTAL FEATURES
Zhipeng Xie*, Yahe Li (* Equal Contribution)
School of Computer Science, Fudan University, Shanghai 200434, China x ciezpé fudan. eeu.. en, yaheli21 @mm.mm.1..mmmm fudan. edu. cn
ABSTRACT
The information bottleneck principle provides an information-theoreiic method for learning a good representation as a trade-off between conciseness and predic- tive ability, which can reduce information redundancy, eliminate irrelevant and superfluous features, and thus enhance the in-domain generalizability. However, in low-resource or out-of-domain scenarios where the assumption of i.i.d does not necessarily hold true, superfluous (or redundant) relevant features may be supple- mental to the mainline features of the model, and be beneficial in making pre- diction for test dataset with distribution shift. Therefore, instead of squeezing the input information by information bottleneck, we propose to keep as much relevant information possible in use for making predictions. A three-stage supervised learning framework is designed and implemented to jointly learn the mainline and supplemental features, relieving supplemental features from the suppression of mainline features. Extensive experiments have shown that the learned repre- sentations of our method have good in-domain and out-of-domain generalization abilities, especially in low-resource cases.
INTRODUCTION
Deep neural network (DNN) has revolutionized variety of fields (inclusive of but not limited to computer vision, natural language processing and speech recognition) in supervised learning tasks (Alam et al. 2020) Otter et al. 2020). Although usually overparameterized, DNN has shown good generalization ability and performs well on test data. Dating back to the Occam's 's Razor philo- sophical principle, it has long been believed that a good model should be as simple as possible. Some research works (Shwartz-Ziv & Tishby 2017 Neyshabur 2017) have been done to uncover the implicit regularization phenomenon or mechanism in the training process of DNNs.
To reveal the dynamics of DNN training process, by visualizing information plane, it was ob- served (Shwartz-Ziv & Tishby 2017) that most of the DNN training efforts are spent on compressing the input to concise representation. This observation has justified the application of the information bottleneck principle (Tishby et al. 1999;; Tishby & Zaslavsky 2015) in the supervised learning set- tings, seeking to capture and efficienlly represent the relevant information in the input variable about good presentation in terms of a funddaennal trade-off between
the label variable and building conciseness and good predictive ability. By explicitly minimizing the mutual information between the input data and its representation and simultaneously maximizing the mutual information between the representation and the label, it has been shown the information bottleneck principle has led to the robustness of fitted model (Alemi et al. 2017).
On the other hand, in the setting of unsupervised representation learning, there is no supervised label to identify relevant irrelevant information, and thus the InfoMax principle (Linsker 1988) is usually used to maximize the mutual information between the input and its representation (Oord et al. 2018: Hjelm et al. 2019), with the expectation that all the (potentially) predictive information is preserved for various downstream supervised tasks.
In contrast to the information bottleneck principle that ignores as many details of the input, our work goes to another end, with the so-called Information Retention principle as our main thrust: when making predictions (or decisions), it is preferable keep as much relevant information as

---

Published as a conference paper at ICLR 2024
possible in use. In other words, the principle of information retention differs from the informa- tion bottleneck: information retention explicitly preserves the relevant redundant information, but information bottleneck implicitly suppresses the redundant relevant information. Bearing the idea of information retention mind, we design and implement a three-stage process for supervised learning. It firstly learns mainline features via vanilla DNN model, then erases salient input features (with respect to the mainline features) from the original inputs to produce modified inputs, and fi- nally uses conditional mutual information as regularization term forcing the supplemental features to be complementary to the mainline features and alleviating feature suppression.
In addition, our approach is also different from the InfoMax principle. It is infeasible to directly maximize the mutual information between the representation and the input in supervised learning, because will learn irrelevant information and produce irrelevant features without any restriction.
Our contribution is three folds:
We propose the information retention principle that favors using as much relevant informa- tion as possible in supervised learning.
To alleviate the problem of feature suppression, we develop three-stage process for infor- mation retention via learning supplemental features.
Experimental results indicate that the learned representation of our method has better in- domain and out-of-domain generalization ability than several competitors.
To better motivate the idea of information retention, also provide our thought from the perspec- tive of causal diagram (Appendix D.I) and investigate a simple toy example (Appendix(D.))
2
METHOD
This section focuses on the details of the proposed method that is called "Information Retention by Learning Supplemental Features" (or InfoR-LSF in short). The basic idea to build up a more abundant supervised representation for classification task. To facilitate discussion, this paper as- sumes that DNN-based classification model consists of two parts: the first is an encoder that maps an input vector E RM to representation vector RD, and the second is a classification head that maps the representation vector z to the final prediction y. Finally, the L(ý, y) denote the super- vised loss used to train the model, where y is the true label of x. In our method, the representation vector z = P ss] E RRD is divided into two groups: the mainline representation 2 E R/2 and the supplemental representation 25 R/22 which are learned in three-stage process. The first stage can be thought of as a "burn-in" phase, initially building up the mainline representation from the original train data. The second stage erases the salient input features from each original input and produces a modified input, where the salient input features denote the features that are important with respect to the mainline representation Z The third stage aims to jointly learn mainline repre- sentation zM and supplemental representation 3y by forcing 2ş to forget the salient input features that have already been captured by mainline representation and alleviate feature suppression, with the help of the modified inputs from the second stage. In the following sections, we will specifically discuss each stage below and also provide an algorithm table in Appendix
2.1 THE FIRST STAGE: INITIAL TRAINING OF MAINLINE FEATURES
At the first stage, the task is to train initial encoder and obtain the mainline features zM by maximizing the mutual information between zM and the label y and (optionally) simultancously minimizing the mutual information between ZM and input x. Therefore, the first-stage objective is:
maximize
[(zMiy) - 3- (Z
(1)
where 33 is the coefficient used control information compression. Setting B 0 will disable the information bottleneck mechanism.
Figure I shows the architecture of the first stage for initially training of our method. The encoding network consists of a backbone fo(-) and a variational encoder (- with few parameters. For input x, we use encoding network to obtain the distribution parameters y and Σ, and generate
2

---

Published as a conference paper at ICLR 2024
maximize T(zM;y) ( (2
backbone network fo(-)
Variational Encoder 9a(-)
yx,
J.. Σ
BM NNH, 2)
Figure I: The architecture of the first stage for initially training the mainline features.
the mainline representation 2M through reparameterization trick (Kingma & Welling 2013). It should be highlighted that in the first stage, we only calculate the supervised loss with information bottleneck restriction on the mainline representation.
2.2 THE SECOND STAGE: SALIENCY ERASING FROM INPUTS
The objective of the second stage is to find and erase salient input features with respect to mainline features ZM from input x and produce a modified input which will be used in the third stage for auxiliary loss calculation. Formally, we denote Xsf as the most salient features that Z has learned in x, and utilize x' MASK(x) = x/x., represent the modified input after removing xat from x, where M-SK(-) refers to feature erasure operation corresponding to the data type at hand.
Salient Input Feature Selection. Given an input data x, its mainline features can be obtained as ZM I6 (x)) via the well-trained mainline representation extractor ga (-)))-)) of the first stage. The task here to find feature subset Xsf C x that the current model relies on most to make prediction. To measure the importance of input features, we use the norm of the loss gradient with respect to input features.
X
topK ([ TEx
The underlying premise is that the larger the loss gradient, the higher the dependence level. Previous studies also supports the rationality of gradient-based feature significance analysis. (Samek al.)
2017
Sun et al. 2021)
Salient Input Feature Erasing. After selecting the salient input features x,f that the current main- line representation Zar most heavily depends on, the next step is to perform MASK(-) operation on the raw input x. For image data, the masking operation works at the level of image patch and Xst designates certain proportion of image patches based on the Equation 2 We then fill these patches with random values that conform to a Gaussian distribution N N(H, o), where the μ and a are calcu- ensure that
lated based on empirical distribution of pixels of the whole training set. Additionally, the padded pixel values are within the range of valid image through the clip operation.
As for text data, each raw input is represented operation is performed at token level. As we utilize pretrained language models with special tokens, erase text feature by replacing certain proportion of tokens selected by gradient norm of token embeddings with [MASK] token.
sequence of tokens, and thus the masking
Additionally, it should be emphasized that the modification x ought to be kept within restricted range, as we expect that the remaining part still contains relevant information about The model could be negatively affected by excessive crasure since the obtained x' may lose excessive supervi- sory information.
2.3 THE THIRD STAGE: JOINT TRAINING OF MAINLINE AND SUPPLEMENTAL FEATURES
In the third stage, the task is to jointly train the whole model and simultaneously learn the main- line features zm and the supplemental features Zs. The mainline features zm still uses the same supervised objective as the one used in the first stage (Equation [.. while the zs needs to suppress learning salient features X,f of zs while learning supervised objective.
The intuitive idea here to force the model to continuously acquire new potential features distinct from the features already leart by current representation ZM, and expect these new features to be helpful for classification. We refer to these features as supplementary features, and we think the key characteristic of these features is that they are partially non-overlapping features with respect to x
3

---

Published as a conference paper at ICLR 2024
maximize I(zyyy) - B. [(Zarix)
(x, y)
backbone network fu(-)
Varistional Encoder ga(+)
(uN,, EM)
Bu ~ Mius E)
shared
shared
backbone
h
Variational
(ug, Es)
29
~ Niss Es)
network fu(-)
h
Enooder go(-)
(1'g, 2''s
Zig ~ N ( 2's)
maximize
[( - B (((ss);)) I(zs;x(x')
Figure 2: The architecture and information flow of the third stage for joint learning of mainline and supplemental features.
and are easy to be suppressed or overlooked by the mainline features zM. possibly weaker harder, as long as they will contribute to the supervisory task.
To accomplish this, we leverage another supplementary representation zy learn these supplemen- tary features. However, it is not guaranteed that zg won't extract the same salient feature Xsr that ZM has already learnt if there are no constraints on Zg. As result, zş should be restricted not to learn features that exists in x but not in x'. According to the chain rule of mutual information, the information nş contains about x could be divided into two parts(the proof can be found at Appendix B.I:
I (zs; x) = [ x') + (Zs;x|xx)
(3)
The first term represents the information 2ş contains about x', and the second term intuitively repre- sents the information zs contains which is unique to x and is not predictable by observing x'. This term exactly what we tend to suppress as we only expect 2ş to learn information relevant to y from x'. In this way, we can derive the learning objective of 2g as follow:
maximize I(2s;y) 3 -(Zz;;;) a [(2g;x(x')
where the the first two terms represent supervised loss with information bottleneck, which is consis- tent with z 's learning objective. The third item is a regularization restriction on Zş representation aiming to force it to learn supplementary features.
Figure 2 illustrates the architecture and information flow of the third stage. We use two different variational encoder 96 and Iv generate zM and zs, respectively. However, the backbone is shared, and the variational encoder only has very few parameters, so our method only uses a small number of additional parameters, which is different from model integration. Additionally, although the joint training stage makes use of the modified inputs generated by the second stage, their use in our method essentially different from the other data-auggmentttion-be based methods. It should be noted that the supervision loss for Zş is still calculated on the representation corresponding to the original inputs but not the modified inputs x'. The modified inputs x' are used only for regularization term [ x Therefore, in contrast to data siitt--agmettation-baaae need not worry about the incorrect supervision signal (or wrong label) that are possibly introduced to the modified inputs x'.
methods, we
2.4 MODEL ARCHITECTURE AND LOSS FUNCTION
In this section, we specifically discuss how to achieve optimization goals proposed above in practice. From the perspective of architecture, we design network with two different variational encoder head on a shared backbone. It should be noted that in inference stage, we will use both ZM and z by averaging logits.
As for the optimizing objectives of each stage, firstly, for the Equation the loss could be derived as follow based on the variational estimate proposed by Allmi al.)
L (x, 2Mz 0, EE [E 2a pe..r (xx/)x)) log ((vl2<())))))) [2))] ) B (2n/ (5)
4

---

Published as a conference paper at ICLR 2024
where (z(a)s) is prior distribution of zM- We use parameterized Gaussian distribution (H Es) to represent (zar). As for [ x|x') used Equation 4 we restrict the term by minimizing its upper bound (the proof can be found at Appendix [B2
Lis Ex [K.. |Po.
(6)
In this way, the total loss the third stage can derived as
== LyIB(x, ZM. 0, Φ) + LVIB(x, zs, 0, 1) + a Lis
(7)
where is coefficient that controls the weight of information suppression. Besides, since the distribution both zM and us follows a Gaussian distribution, there exists a closed form solution for calculating KL-divergence in the above losses.
3 EXPERIMENTS
This section is devoted to a thorough empirical study of the proposed InfoR-LSF method" We first evaluate it in-domain generalization ability on classification and regression tasks (Section B.1] and then investigate the out-of-domain generalization of learned representation
sentiment analysis task (Section 3.2 addition, we also conduct experiments to analyze the effect of information retention (Section 3.3 and study the hyperparameter sensitivity (Section 3.4) Finally, results of ablation experiments are presented in Section 3.5
Architectures. We choose ResNet-18 (He et al. 2016) as the backbone for image classification, the pretrained BERT BASE (Devlin et 2018) as the backbone for text-related tasks, and MLP network as the backbone for tabular regression. More dataset information and implementation details about training
and optimizer are provided in the Appendix A.I and Appendix
Baselines. We compare against several influential works: IFM (Robinson et al., 2021). a method which avoids shortcut solutions by implicit feature modification; FGSM (Goodfellw et al. 2014), classic adversarial training method in computer vision; VIB (Alemi et al. 2017). variational ap- proximation to the information bottleneck by leveraging the reparameterization trick; VIBERT (Ma- habadi et al. 2021), method implementing the variational information bottleneck the pretrained BERT to suppress irrelevant features and enhance generalization ability when fine-tuning. In addi- tion, the basic R8sNet-18 (or BERT BASE) model serves as a baseline.
3.1 IN-DOMAIN GENERALIZATION ON SUPERVISED LEARNING TASK
Table I: CIFARIO classification task accuracy under different train data size.
Train Data Size
Model
50
100
200
500
1000
2000
3000
50000
ResNet-13 18
17.2
22.6
31.1
40.4
48.9
63.3
74.2
95.1
IFM
17.1
22.4
31.5
42.1
51.8
65.8
75.1
94.6
FGSM
20.1
23.7
31.4
40.3
47.7
58.1
65.5
91.8
VIB
18.6
22.4
31.0
39.7
49.9
64.8
74.7
95.1
InfoR-LSF
20.3
24.5
32.1
42.1
52.8
67.3
76.2
95.2
+3.1
+1.9
+1.0
+1.7
+3.9
+4.0
+2.0
+0.1
Image classification. We use two image classification datasets, CIFARIO (Krizhevsky et al. 2009) and CIFAR Krizhevvvykkk al.) 2009). Different sizes of training data (subsampled from the original training set), ranging from 50 to 50000, are used to evaluate the test accuracy in low- resource settings. Each experiment is conducted three times, each time with different random seed for data sampling. The average test accuracy for CIFAR10 are shown Table The performance on CIFAR100 provided in Appendix A.3
Text classification. We use two sentiment analysis datasets, namely IMDB Maas et al. 2011 and YELP /Zhang al. 2015). The sizes of training data range from 50 to 1000, with training examples
"Code available wtwphhtt://///t/ http://.......oooo...ooommo http://...oo////ahaeeeeaa yaaee/ InfoR-1.s5
5

---

Published as a conference paper at ICLR 2024
Table 2: Text classification task accuracy under different train data size.
Train Data Size
Dataset
Model
50
100
200
500
1000
BERT
66.6 (2.2)
77.9 (2.3)
85.6 (0.5)
87.1 (0.6)
88.7 (0.3)
IFM
66.1 (2.2)
78.2 (2.4)
85.6 (0.7)
(0.7)
88.7 (0.4)
IMDB
VIBERT
68.9 (2.5)
80.8 (1.7)
86.1 (0.6)
87.8 (0.7)
88.8 (0.4)
InfoR-LSF
75.5 (2.3)
83.0 (2.9)
86.9 (0.4)
88.3 (0.5)
89.4 (0.4)
A
+8.9
+5.1
+1.3
+1.2
+0.7
BERT
35.1 (1.8)
39.6 (2.1)
43.1 (1.7)
51.9 (0.9)
55.6 (0.7)
IFM
35.7 (2.5)
40.1 (1.8)
(1.0)
50.9 (1.0)
55.5 (0.7)
YELP
VIBERT
37.7 (1.2)
40.8 (2.3)
44.8 (2.2)
(2.1)
55.4 (0.6)
InfoR-LSF
39.6 (1.1)
41.4 (1.4)
44.9 (2.4)
53.6 (0.6)
55.9 (0.3)
A
+4.5
+1.8
+1.8
+1.7
+0.3
randomly sampled with five seeds (13, 21, 42, 87 and 100). We report the average and standard deviation test accuracy Table 2 Here, under low resource settings, we do not use the original validation set, but instead sample validation subset of the same size the train data.
Table 3: STS-B test set Pearson correlation coefficient under different train data sizes.
Train Data Size
Dataset
Model
50
100
200
500
1000
BERT
72.2 (3.2)
79.1 (1.9)
83.8 (0.8)
86.4 (1.0)
87.5 (0.2)
IFM
(3.1)
79.2 (1.9)
84.0 (0.9)
86.8 (0.7)
87.6 (0.2)
STS-B
VIBERT
74.4 (2.8)
81.9 (1.8)
85.0 (0.4)
87.1 (0.3)
88.4 (0.3)
InfoR-LSE
82.4 (2.0)
85.4 (0.5)
87.5 (0.6)
88.7 (0.3)
A
+2.8
+3.3
+1.6
+1.1
+1.2
Textual Similarity Score Regression, The task STS-B(Cer et al. 2017) is to regress textual simi- larity score. The results of STS-B are shown in Table
Traditional Tabular Regression. We further conduct experiments on tabular regression task Appliance Energy Prediction (AEP) (Can-
Table 4: Coefficient of determination R²) of AEP under different train data sizes.
danedo (017) from UCI Machine Learning Repository (Asuncion & Newman 2007). We process the data as same as public preprocess- ing and and then apply 5-layer MLP for predic- tion. We adjust the hyper-parameters to make MLP reach the public SOTA performance, and then verify the effect of InfoR-LSF on this ba- sis. The results of energy prediction are shown in Table
Train Data Size
Model
10%
20%
50%
100%
MLP
0.338
0.456
0.597
0.684
IFM
0.373
0.469
0.605
0,680
VIB
0.347
0.471
0.602
0.679
InfoR-LSF
0.376
0.483
0.618
0.691
A
+0.038
+0.027
+0.021
+0.007
Observation and Analysis. Firstly, from Ta-
ble D and Table 2. obvious that our InfoR-LSF method surpasses all competitors under all settings of training data sizes, for both image and text classification tasks. Table and ther prove that InfoR-LSF also performs well on regression tasks. These observations indicate the universality of information retention principle. We attribute this substantial performance gain its ability of learning diverse features, including not only the mainline features but also the supplemen- tal features (note that the supplemental features may be redundant with respect the mainline ones from the perspective of training data). Secondly, VIB achieves little performance gain on CIFAR, while VIBERT exhibits more performance gain on text-related tasks. We attribute this the essen- tial difference between images and texts: the tokens, as the basic units of texts, are meaningful, and
and Table 4 fur-
https: : www. www.wwaaggiin kaggle, e. wtt. www.///od//rreeeeeeee
accuracy-99

---

Published as a conference paper at ICLR 2024
only few of them indicate the label of given text; however, the pixels, the basic units of images, are meaningless alone, over-simple features that comprise fewer pixels may not necessarily imply better generalization ability. That also explains why VIB is more often applied to NLP tasks. Last but not the least, InfoR-LSF exhibits much notable improvements low resource conditions, and its performance gain gradually declines as the number of training examples rises. It suggests that the feature redundancy may diminish with the increased availability of labeled data resource, and the train and test sets tend be more equally-distribute as the data size grows.
3.2 OUT-OF-DOMAIN PERFORMANCE
Since InfoR- is wedded to learn more versatile relevant features, it is naturally hypothesized that the learned representation beneficial to similar out-of-domain tasks. To verify this hypothesis. we choose the sentiment classification task, and use the full-size YELP data (Zhang et al., (015) as the source domain to train models. By freezing the backbone and retraining a linear tssk-ipecific classification head, we evaluate the linear readout of each model series of out-of-domain target tasks, including IMDB (Maas et al. (201T). YELP-2 (Zhang et al. 2015), SST-2 (Socher et al. 2013), SST-5 (Socher et al. 2013). MR (Pang & Lee 2005). Amazon-2 (Zhang et al. 2015) and Amazon- 5 (Zhang et al. 2015). Each linear head is traindd with 1000 labeled data from the target task. As
all target tasks, InfoR-LSF consistently achieves the highest improvement. We conjecture the reason to be our method's ability of extracting more versatile features and thus the learned representation is more likely cover the useful features in target domains, leading to better out-of-domain generalization.
shown in Table
Table 5: Test accuracy of models transferring to new target datasets. All models are trained on YELP and evaluated linear readout on the target datasets. A are the absolute differences with BERT.
Target Dataset
Model
YELP
YELP-2
IMDB
SST-2
SST-5
MR
Amazon-2
Amazon-5
BERT
65,81
94.95
88.24
86.54
44.88
80.70
81.59
54.53
VIBERT
66.00
95.87
88.05
83.90
44.75
81.20
81.81
56.05
InfoR-LSS
66.31
95.89
88.55
88.19
46.28
82.00
83.03
57.43
A
+0.5
+0.94
+0.31
+1.65
+1,4
+1.3
+1.44
+2.9
3.3
EFFECT VERIFICATION OF INFORMATION RETENTION
The main motivation of our method is extract much information about y from as possible. Previous experiments have demonstrated its superiority in various downstream tasks, but is the im- provement in effect due the idea of information retention? To explore this question, we investigate whether the model exploits more input features by observing the model's attention distribution on input images. If the learned representation at the final hidden layer utilizes more original input fea- tures, the attention of the model should be distributed more evenly more pixels. Otherwise, the attention will be distributed intensively on fewer pixels. Therefore, we use the gradient norm of loss as the model's attention score for each pixel, which has been proven to be effective method for evaluating the contribution of features (Simonyan et al.) 2013 Zeiler & Fergus) 2014). Specifically, first obtain the gradient V.L(z.y) y) for each pixel E x, normalize the absolute values of the gradients over the entire image, and then examine the distribution of gradient amplitudes across the entire test From the visualized results in Figure 3 it can be observed that our method has the most dispersed attention distribution on the original input x under different training data sizes, and its distribution peak is significantly lower than other methods. On the contrary, VIB has the highest peak of gradient distribution, and thus attends intensively on the fewest pixels. The phenomenon meets our expectations, as the VIB restricts the mutual information between the representation and input x through an explicit regularization term, so the utilization of information by the repre- sentation z on x is insufficient. Surprisingly, The gradient distribution of FGSM, which is based on adversarial training, is actually more concentrated than baseline, indicating that adversarial attacks can improve the robustness of the model, but may not necessarily force the model to leverage more original input information. In summary, compared to other baselines. method exhibits a more uniform and dispersed trend in pixel level attention distribution, demonstrating the mechanism of information retention is in work effectively.
7

---

Published as a conference paper at ICLR 2024
Beseline
Baseline
Baseine
IM
IPM
IPM
FGSM
FCSM
FGSM
Clurs
244
206
Tradlon
dedten mocia take
SROW MMX Cradtc Aeator
200 train samples
(b) 500 train samples
1000 train samples
Figure 3: Gradient amplitude distribution over test set of different method on CIFAR1O.
3.4 SENSITIVITY ANALYSIS OF AND MASKING RATIO
To examine the effect of loss term Lis in Equation We perform sensitivity analysis experiments by varying coefficient a and masking ratio on the IMDB dataset under low resource setting (50 train data). As shown in Figure 4a (or 4b). test and validation accuracy first increase and then decrease by increasing a (or by increasing masking ratio), and they both achieve the best performance at = 1e---- 3 (or with the masking ratio of 0.05). means that over-high values of a and making ratio excessive restriction 2ş representation, and thus do harm to the performance.
Figure(Fojshows the difference in the attention distribution of mg and Z on salient input features X.t- We calculate the attention proportion by first calculating the gradient norm V[x,, y))l and ||Vx, [(zs. y)ll. and then normalizing them across the whole sentence. Obviously, the gap between regularization terms are tightening. Additionally, we have noticed that an excessively large a will also constrain ZM meanwhile because MM and 2y share the encoder parameter. This can also explain why excessive harmful.
zs and nş gradually increases, indicating that the restrictions
Attention Proportion
Accuracy Val dotion Accuracy
24
Accuracy
Validation Accuracy
041
5/2
masking rallo
(a) Accuracy vs a
(b) Accuracy vs masking ratio
(c) Attention proportion on salient features Xst of zM and z
Figure 4: Sensitivity analysis of and masking ratio on IMDB dataset.
3.5 ABLATION STUDY
To explore the effect of the regularization restriction zş and the IB restriction, we conduct abla- tion experiments on three datasets. We remove the regularization restriction on 2g by setting a = and remove IB restriction by setting 8
The results are shown Table 6 Firstly, regard- less of whether there are IB restrictions, the performance does reduce on all considered datasets if 0, demonstrating the solid gain of the regularization term of our method. Secondly, we find that in some cases(especially on CIFAR 10), only removing the IB restriction( mance improvement, indicating that the IB restriction can lead to negative effects some cases. The phenomenon is also consistent with our previous analysis in Section 3.J that the gains brought by IB on text are more significant than that on images. Furthermore, it should be noted that the ablation results still differ from the base model due to the of two variational encoders. When 0, InfoR-LSF degenerates to a VIB model with two heads (the modified input x' will not participate in any loss calculation if 0). In this case, the results should be similar VIB VIBERT.
0) results in perfor-
8

---

Published as a conference paper at ICLR 2024
Table 6: Average ablation results over runs under different data size of base model, InfoR-LSF and InfoR-LSF without regularization restrictions
Train Data Size
Data
Model
50
100
200
500
1000
BERT
66.6 (2.2)
77.9 (2.3)
85.6 (0.5)
87.1 (0.6)
88.7 (0.3)
InfoR-LSF
75.5 (2.3)
83.0 (2.9)
86.9 (0.4)
88.3 (0.5)
89.4 (0.4)
IMDB
InfoR (F)
(3.1)
81.1 (3.9)
86.0 (0.9)
87.5 (0.4)
88.8 (0.2)
InfoR-LSF(
(2.8)
30.9
86.4 (0.5)
87.8 (0.4)
89.2 (0.2)
(n008-
(2.7)
(2.6)
85.9 (0.9)
87.5 (0.5)
88.6 (0.6)
BERT
(1.8
(2.1)
43.1 (1.7)
51.9 (0.9)
50.6
InfoR- LSF
39.6
(1.4)
44.9 (2.4)
53.6 (0.6)
55.9(0.3)
YELP
InfoR (SF((=0)
(2.1)
41.0 (1.4)
53.3(0.7) (2.1)
55.2 (0.9)
InfoR-LSF($300)
(2.5)
41.0 (3.6)
45.6 (1.9)
53.5
55.7 (0.5)
InfoR-LSF(O=0,3-0
(1.7)
40.1 (2.5)
(2.3)
51.3 (0.7)
55.1 (0.3)
ResNet
17.2(2.1)
22.6(1.1)
31,((16)
40,4(1.6)
48.9(1.2)
InfoR-LSF
20.3(1.5)
24.5(0.2)
32.1(1.8)
42.1(08))
52.8(10)
CIFARIO
InfoR-LSF(a=0)
19.2(0.8)
22.5(0.5)
30.4(188
41.0(0.4)
51.1(0.9)
InfoR-LSF(3E0)
20.4(1.6)
23.8(0.9)
318(2.0)
42.7(0.8)
52.1(19)
InfoR-LSF(Cn -SF(0=0,8=0)
18.7(2.2)
21.5(0.4)
30.6(1.6)
41.5(0.7)
50.4(1.5)
4
RELATED WORK
Information Bottleneck. The information bottleneck(1I)) (Tishby et al. 1999) was first proposed in traditional machine learning and then used to analyze and interpret the behavior of deep neural networks (Tishby & Zaslavsky 22055 Shwartz-Ziv & Tishby 2017). Later, Alemi et al. (2017) pre-
sented variational information bottleneck (VIB) to improve the learning of DNNs by optimizing a variational bound of IB objective. (Federici et al., 2020) extended the IB method learn robust rep-
resentations in unsupervised multi-view setting by minimizing superfluous information not shared by the two views. Additionally, IB were introduced NLP tasks such dependency parsing (Li &
Eisner 2019) and unsupervised sentence summarization (West et al. 2019). Mahabadi et al. (202))
applied VIB in the finetuning of pretrained BERT (Devlin al. 2018 showing good low resource
performance. Besides, IB also helped domain generalization (Ahuja et al. 2021 Du et al. 2020).
Information Maximization. The InfoMax principle (Linsker 1988 Bell & Sejnowski 1995) were
proposed to advocate maximizing mutual information between input and output. Hjeel et al. (2019) applied InfoMax principle on unsupervised representations learning for deep neural networks and Bachman al. (2019) further developed self-supervised representation learning based on maximiz- ing mutual information between features extracted from multiple views. Besides, the InfoMax prin- ciple has also been leveraged in contrastive representation learning, such contrastive predictive coding (Oord et al. 2018; Henaff) (020) and contrastive multi-view coding (Tian et al. 2020).
Although it has been observed that IB methods can learn parsimonious features with better general- ization in resource-liminn limited scenarios, they implicitly suppress other redundant relevant features. The main motive of this work advocates the learned representations should incorporate these suppressed relevant features, which makes it distinct from IB methods.
CONCLUSION
In this work, we introduce the principle of information retention, which aims keep as much relevant information possible in use for making predictions. We further design three-stage supervised learning framework named InfoR-LSF for information retention by jointly learning the mainline and supplemental features. In experiments, we compare InfoR-LSF against other methods, and its strong performance in different fields under both low-resource and out-of-domain scenario shows that InfoR-LSF can be practically applied to various type of tasks. Furthermore, analysis experiments indicate that our framework indeed achieves information retention and extracts more relevant features than other competitors.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is financially supported by National Natural Science Foundation of China (N0.62070072). We are grateful to the anonymous reviewers for their valuable comments.
REFERENCES
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean- San-CChistoophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization. Advances in Neural Information Processing Systems, 34:3438-3450, 2021.
Mahbubul Alam, Manar D Samad, Lasitha Vidyaratne, Alexander Glandon, and Khan M Iftekharud- din. Survey deep neural networks in speech and vision systems. Neurocomputing, 417:302- 321. 2020.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy, Deep variational information bottleneck. In ICLR, 2017.
Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Anthony J Bell and Terrence J Sejnowski. An information-maxxmizztion approach to blind separa- tion and blind deconvolution. Neural computation, 7(6):1129-115; 1995.
Luis Candanedo. Appliances energy prediction. UCI Machine Learning Repository, 2017. DOI: https//doo.oo/11.2242//ccc/
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task I: Semantic textual aimilarty-multiligene and cross- cooss-lingual focused evaluation. arXiv preprint arXiv:1708.0000 2017.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1rXiv:1810.04805. 2018.
Yingjun Du, Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees GM Snock, and Ling Shao. Learning to learn with variational information bottleneck for domain generalization. Computer Vision-ECCT ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16. pp. 200-216. Springer, 2020.
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata, Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07017, 2020.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: arXiv: 1412.6572, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International conference on machine learning, pp. 4182 4192. PMLR, 2020.
R. Devon Hjelm, Alex Fedorov. Samuel Lavoie-Marchildon. Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representattions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019. URL https:
//openreview net./ fmm d=Bk1
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6 424269800 2014.
10

---

Published as a conference paper at ICLR 2024
Diederik P Kingma and Max Welling, arXiv:13/2.614,222 2013.
Auto-ennoding variational bayes. arXiv preprint
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Xiang Lisa Li and Jason Eisner. Specializing word embeddings (for parsing) by information bottle- neck. arXiv preprint arXiv:/910.00163 2019.
Ralph Linsker. Self-organization in perceptual network. Computer, 21(3):105-117, 1988.
Andrew Maas, Raymond E Daly, Peter Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. Variational information bottle- neck for effective low-resource fine-tuning, arXiv preprint arXiv:2106.0544 2021.
Behnam Neyshabur. Implicit regularization in deep learning. arXiv preprint arXiv:1709.01920 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 180..0:80 2018
Daniel W Otter, Julian R Medina, and Jugal K Kalita. A survey of the usages of deep learning for natural language processing. IEEE transactions neural networks and learning systems, 32(2): 604-624, 2020.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect rating scales. arXiv preprint cs/0506075, 2005.
Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shorteut solutions? Advances in neural information processing systems, 34:4974-4986, 2021.
Wojeiech Samek, Thomas Wiegand, and Klaus-Robert Müller. Explainable artificial intelli- gence: Understanding, visualizing and interpreting deep learning models. arXiv preprint 1788v:1708.08296, 2017.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint T.d3:/770300880 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.603: 2013.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013..
2013.
Xiaofei Sun, Diyi Yang, Xiaoya Li, Tianwei Zhang, Yuxian Meng, Han Qiu, Guoyin Wang, Eduard Hovy, and Jiwei Li. Interpreting deep learning models in natural language processing: A review. arXiv preprint arXiv:2110. 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16. pp. 776-794. Springer, 2020.
Naftali Tishby and Noga Zaslavsky.
Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1-5. IEFF, 2015.
Naftali Tishby, Fernando C. N. Pereira, and William Bialek. The information bottleneck method. In The 37th annual Allerton Conference on Communication, Control, and Computing, pp. 368-377, 1999.
11

---

Published as a conference paper at ICLR 2024
Peter West, Ari Holtzman, Jan Buys, and Yejin Choi. Bottlesum: Unsupervised and self- supervised sentence summarization using the information bottleneck principle. arXiv preprint arXiv:1909.07405. 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014; 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp. 818 833.. Springer, 2014.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
12

---

Published as a conference paper at ICLR 2024
A
EXPERIMENTAL SUPPLEMENT
A.1 DATASET STATISTICS
Table 7 shows statistic information of datasets used in our experiments.
Table 7: Datasets used in experiments.
Dataset
#Lables
Train
Valid
Test
Image Classification
CIFARIO
10
50K
10K
CIFAR100
10
50K
10K
Sentiment Classification
IMDB
2
20K
5K
25K
YELP
62.5K
7.8K
8.7K
YELP-2
560K
38K
SST-2
2
6.9K
0.9K
1.8K
SST-5
5
8.5K
1.1K
2.2K
MR
8.7K
2K
Amazon-2
3600K
400k
Amazon-5
3000K
650K
Semantic Textual Similarity
STS-B
5.8K
1.5K
1.4K
Regression
Appliance Energy Prediction
15.8K
3.9K
A.2 IMPLEMENTATION DETAILS
For CIFARIO, refering to the open source implementation") we train the model for 200 epochs with batch size 256, initial learning rate 0.1 and weight decay 5c-4. We apply SGD optimizer with momentum 0.9 and a step scheduler that decays the learning rate by 0.1 every 160 epochs. For CIFAR100, we train the model for 240 epochs with batch size 64, initial learning rate 0.05 and weight decay 5e-4. We use SGD optimizer with momentum 0.9 and multi-step scheduler that decays the learning rate by at milestones 150, 180, 210. For text classification tasks, we fine- tune all the models for 25 epochs with batch size 8 to allow them to converge. We use Adam W optimizer (Kingma & Ba 2014) with initial learning rate 2c-5 and warmup ratio 0.05 for all models for fine-tuming. Besides, of the validation set.
early stopping for in all text experiments based on the performance
As for hyper-paanee peeemmaaaeetess of variational information bottleneck, hidden layer representation dimension, i.e., 384 for BERT and 256 for ResNet-18. And the of in- formation bottleneck is set le-5 for all experiments. We use sample size for reparameterization.
bottleneck size as half of the
As for the coefficient a of our method, we use 1e-3 for all the text experiments, 1e-2 of CIFAR10 and CIFAR100 and 0.1 for Appliance Energy Prediction. And for the modification of input features, in all our experiments, the mask proportion is 5%, i.c., 5% of the patches in the image or 5% of the tokens for the sentence. In addition, for both CIFARIO and CIFARI00, the patch size used when selecting salient features is 4 4, i.e., each image is divided into 8 x 8 image patches.
A.3 IMAGE CLASSIFICATION PERFORMANCE ON CIFAR
On CIFAR100, construct low resource subset of various sizes from original train set, with scales ranging from 1000 to 50000. Results are provided in Table g Consistent with the conclusion on CIFAR10, method outperforms all baseline methods at all data sizes.
far
13

---

Published as a conference paper at ICLR 2024
Table 8: CIFARI00 classification task accuracy under different train data size.
Train Data Size
Model
1000
2000
3000
5000
10000
20000
50000
ResNet- 18
13.90
20.65
27.10
38.08
55.52
67.14
77.85
IFM
14.04
21.71
28.46
39.46
56.72
67.19
77.53
FGSM
14.19
20.56
26.21
34.80
48.46
59.60
71.66
VIB
13,94
21.17
27.85
39.46
56.30
67.30
77.54
InfoR-LSF
15.51
22.61
30.43
43.32
58.79
68.85
78.44
A
+1.61
+1.96
+3.33
+5.24
+3.27
+1.71
+0.59
A.4 COMPUTATIONAL COMPLEXITY ANALYSIS
Theoretically, MORF-LSF conducts forward propagation and backward propagation twice during the training phase, and there is double data in the second time. As result, the time consumption should be approximately 2-3 times that of the base model. During the inference phase, InfoR-LSF doesn't t require additional backward propagation. The increase in calculation amount only comes from the additional variational encoder. Therefore, the time cost should be comparable to the base model. We measure the running time of each method Table 9 in which the base model is used as benchmark. It can be observed that the training time of infoR-LSE is about 2.43 times that of the base model, and the inference time is about 1.03 times that of the base model, which is consistent with the conclusion of the theoretical analysis.
CIFARIO and IMDB. The results are shown in
Table 9: Computational complexity analysis of each method. A % are relative differences with base model(ResNet-Ill BERT).
Dataset
Model
Train Time
A
Inference Time
A
baseline
425s
1.98s
IFM
430s
1.18%
2.02s
1.97%
CIFARIO
FGSM
859s
102.12%
1.99s
0.66%
VIB
428s
0.71%
2.02s
2.02%
InfoR-LSH
1033s
143.06%
2.05s
3.28%
baseline
101s
100.56s
IFM
105s
4.16%
101.54s
0.98%
IMDB
VIBERT
108s
6.32%
101.89s
1.32%
InfoR-LSE
243s
139.75%
103,42s
2.84%
B PROOFS
2.4
g/
Figure 5: The relationship between x, x', ZM. Zs, and z's.
In the framework of InfoR-LSF, we first train Z on original input x, and then mask the salient feature of according to zы obtain x'. In the third stage, we jointly train both zm and zg. where zs is restricted by regularization term. The relationship between x, x'. ZMs 2s, and z's is shown Figure 5 It can be derived that x' and 2ş conditionally independent for any given x.
14

---

Published as a conference paper at ICLR 2024
B.1
PROOF OF EQUATION
Hypothesis:
Given x' and ns conditionally independent: [(2g;
Proof.
[(zs;x) [(zs; xx') - [(2s;
[[zs; xx') x(zs:)) + I(2;:((x')
B.2
PROOF OF EQUATION
Hypothesis:
Given x, x' and uy are conditionally independent: p(z5x' (x) = p(zssx))p((()
Proof.
p(zs. x((')
x(2;;((x) = Ex(= pep
log
pl x (osssxx))((x>> (x|xxx))
pl Bs. x xx
Ex xx x/xxxx Bag log
p x )p (x(x')
p(z xx')
= Ex Xxx''=p(xx' Bug log
p p(25 (x)
p(zs| xx)
Ex xxx' 2 (zs~p(as)x) log
p(z,) xx)
p(zs) p)''
Ex,x'=m p()))))) log
p(z's xx' pp(zsxx)
Ex DKL [p(zpp( |x)] xx) pp x')] - Dww (p(zspp/p)pppee (x)ss///))))
I Ex, xxxxxx
[KL (p pzzzs x']]]
C ALGORITHM TABLE OF INFOR-LSF
The framework InfoR-LSF shown in Algorithm
Algorithm InfoR-LSF
Input:
Input data RM and label y. A model M consisting with backbone network fo(+) and two variational encoders 9o (-) and go (-). Training epochs T'. Feature suppression coefficient a and VIB coefficient
Output: Final model M.
1: Train mainline feature = (x())x)) with VIB loss LVIB Z. 0, Φ) at the first epoch.
2: for = 1, ....,
do
topk mex I/ ((C((g(f((()))) y) 4: Erase salient feature x&t from x to obtain erased input x'.
3: Find salient input feature of mainline feature zM
5: Jointly train mainline feature 2M ge (x)( and supplemental feature 25 90 (fo(x)) by
L Lvin(x x, 2M, 0, (. + LVIB (x, 2s, 0, 1b) + a Lis
6: end for
7: return Final model M
D MOTIVATIONS FOR INFORMATION RETENTION PRINCIPLE
D.1 FROM THE PERSPECTIVE OF CAUSAL DIAGRAM
15

---

Published as a conference paper at ICLR 2024
To motivate the information retention principle. let us investigate the causal diagram in Figurel6 Since the common cause h are latent and unob- served, correlation exists between the input x and the label y- Traditional machine learning or deep learning methods usually exploits the correlation make prediction are based on the correlation, with bel for the input data x. At the level of causal mechanism, the task of inferring y from con- sists of two steps: to infer the posterior of latent cause h of the observed input x,and then to pre- dict y based on p(h(x) During this process, taking more imput features of x into consideration is expected to yield better estimation of h, although some components of h may have effect y.
y
h
i = f(x)
x
f(x) as the predicted la-
Figure 6: The causal diagram that assumes that there is common cause of the input x and its real label y
D.2 A TOY EXAMPLE IILUSTRATE THE IDEA INFORMATION RETENTION
Table 10 shows simple classification dataset, where the input consists of four variables {ø1, 22, and the label is a binary variable. We use this exam- ple illustrate the idea of information retention prin- ciple.
Table 10: A simple motivating example
33
y/
2
True
1
2
True
The label y can be perfectly predicted by using the fea- ture fi 1i + 22, because fi 2 implies y = True, and S1 implies y = False, se. Therefore, the feature fi can be thought of as good representation, which both concise and predictive.
2
2
2
True
2
False
2
2
1
False
1
False
Next, let us have a look the other two simplest fea- tures f2 = Iy and fa = 24, which are also relevant to the label and have medium predictive ability. However, since fi has captured all the information of the label taking /2 or fa into consideration will not bring any lifting in predictive ability but will lead to increased mutual information with the input. Thus, both f2 and fa are suppressed and discarded by the information bottleneck principle.
Finally, it comes the test time and a new data [x] 1, has the feature fi = 4. The classification model based on fi has not seen this feature value in the training phase and cannot make reliable prediction label. However, feature f2 and feature fs can deal with this situation, and both of them supports the prediction of y True.
23 1, 2] arrives, which
Therefore, beneficial and desirable endow with the ability of relieve redundant relevant fea- tures from the suppression of existing mainline features. Such redundant relevant features is supple- mental to the mainline features.
Furthermore, given the model that heavily relies on f1 (called the mainline feature) and thus attends the input features 2 and 22. one possible solution build up some new supplemental features by requiring them capture the least information that has already been captured by mainline features, and simultaneously maximize their mutual information with the label.
E AN EXAMPLE FOR VISUALIZATION OF LEARNED FEATURES
Figure shows an example from IMDB test set. We compute the gradient norms on input token embedding sequences and normalize them over the whole sentence. The three lines respectively represent the gradient distribution of BERT, ZM head and 2s head InfoR-LSF, The depth of the color represents the magnitude of the gradient. It can be observed that with the regularization term Lis, zg partly erases salient features of ("movie a 1 1.") and learns new features(" mada me sick .") which are also helpful for classification. In addition, from the perspective of out-of-domain generalization, "movie a 1 is task-specific feature while "made me sick" is a more general feature, this also explain the better out-of-domain performance of InfoR-LSF..
16

---

Published as a conference paper at ICLR 2024
Z5
ZM
BERT
0055
Figure 7: Visualization of one example on IMDB test set. The depth of the color represents the magnitude of the gradient.
17


Reviews:
[{"review_id": "RQzkWH5EyZ", "reviewer": "Reviewer_RFaU", "summary": "This paper introduces the principle of information retention, which stands somewhat on the opposite side of the principles of Occam’s razor and information bottleneck. It focuses on utilizing as much relevant information as possible in decision making instead of favoring simplest model. Next, this underlying idea get materialized by relieving supplemental features from the suppression of mainline features in a three-stage algorithmic framework. Extensive experimental results have manifested the effectiveness of the proposed method in both image classification and text classification tasks.", "questions": "1) Besides in-domain generalization, I wonder whether the proposed method has better out-of-domain generalization ability in some degree?\n2) To my understanding, the supplemental features are learned by minimizing the conditional mutual information $I(\\mathbf{x}, \\mathbf{z}_S|\\mathbf{x}’)$, is it right? Why not just minimizing the mutual information between $\\mathbf{z}_S$ and $\\mathbf{z}_M$?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) The approach overall is well-motivated and well-described. I like the organizational form of the paper, from conceptual idea to framework design and to concrete implementation, which is easy and natural to follow.\n2) The principle of information retention and the method of learning supplemental features are novel and inspiring. It seems that the approach and method can be readily extended to other tasks.\n3) Experimental results on image classification and text classification tasks have manifested the approach has good in-domain generalization ability, especially in poor-resource settings.", "weaknesses": "1) The method can be better described with an overall workflow or architecture. \n2) The paper needs further refinement of some minor details, and I have found some grammar and spelling errors in the sections of experiments and conclusions.\n3) Although I believe that the proposed approach can benefit out-of-domain generalization,it would have been better if some preliminary experiments had been conducted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kEK3wCTKUg", "reviewer": "Reviewer_BzTN", "summary": "This paper proposes retaining supplemental features during training, as mainline features learned under the information bottleneck principle are insufficient to address out-of-distribution cases.", "questions": "- What contributions can the research in this paper make to the current mainstream deep learning models, such as GPT and Diffusion?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "The illustration of the proposed method is easy to follow.", "weaknesses": "- The paper is unqualified to be a top conference paper in several aspects, including novelty, contribution, and writing quality.\n- The experiment design is outdated as of 2023.\n- The writing needs improvement. For example, in the abstract, the authors mention, \"relevant features may be supplemental to the mainline features of the model.\" However, the subsequent phrase \"to address this problem\" lacks coherence. Additionally, in the conclusion, verb tenses are inconsistent, alternating between \"introduce\" and \"proposed.\"\n- The paper lacks a comprehensive literature review.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L27MgikB0c", "reviewer": "Reviewer_LTAN", "summary": "The paper presents a new model architecture based on the intuition that redundant yet relevant features are also crucial, contradicting the information bottleneck principle. In detail, the authors designed a three-stage training algorithm: firstly, to learn the mainline feature; secondly, to erase the salient input; and thirdly, to learn the supplementary feature. The authors conducted classification tasks in both vision and language, and performed analyses to validate the usage of redundant features.", "questions": "How is Equation (5) and (6) exactly derived?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper starts with the intuition that the information bottleneck principle may be validated under specific situations and then formulates the idea into a training pipeline and model architecture. This is a very exciting buildup from idea to implementation. At the same time, the authors design the whole pipeline in a way that is very close to the initial idea, and they consider the details very thoughtfully. In the analysis part, they also provide sufficient evidence that the redundant yet relevant features are truly learned.", "weaknesses": "1. The presentation can be greatly improved. For instance, out-of-domain scenarios are mentioned in the abstract, but they are not explored in later experiments. The two examples in the introduction are unnecessary; they might even hinder comprehension, as some context is missing (as in example 1.1), and the usage could be restricted by the examples provided.\n2. Section 3.3 shows the proposed algorithm extracts more information than other algorithms. It is also necessary to show that the extracted information is actually relevant information.\n3. It would be helpful to provide some visualization to show that the second stage does erase the salient features. \n4. Some context in information theory is missing. A section of related work is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l2QX9f8u6l", "reviewer": "Reviewer_WVdm", "summary": "The paper presents a novel concept in supervised learning called information retention, emphasizing the importance of utilizing as much relevant information as possible for predictions. The authors introduce the InfoR-LSF framework, a three-stage learning system designed to process both mainline and supplemental features without allowing the mainline features to suppress the supplemental ones. Through experiments on the CIFAR10, IMDB, and YELP datasets, the paper demonstrates that InfoR-LSF outperforms existing methods, especially in low-resource scenarios. This research underscores the potential of InfoR-LSF to be effectively applied across various tasks, highlighting its unique approach to retaining and utilizing information.", "questions": "1. Why were CIFAR10, IMDB, and YELP selected as the benchmark datasets? Are there plans to test the method on more diverse or domain-specific datasets?\n\n2. How generalizable is the InfoR-LSF framework? Can it be easily adapted to other tasks or domains outside of image and text classification?\n\n3. Given that adversarial training methods like FGSM were considered as baselines, did the authors consider evaluating the adversarial robustness of models trained with InfoR-LSF?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The introduction of the concept of information retention and the development of the InfoR-LSF framework offer a fresh perspective on supervised learning. The method's focus on harnessing both mainline and supplemental features without interference distinguishes it from conventional approaches.\n\n2. The paper conducts experiments on multiple datasets spanning different domains (CIFAR10 for image classification and IMDB and YELP for text classification). This extensive evaluation offers credibility to the method's versatility and robustness across varying types of data and tasks.\n\n3. Beyond just presenting a novel method and its performance metrics, the paper delves deep into understanding its functioning. Through experimental verifications, like observing the model's attention distribution on input images, the authors effectively demonstrate the actual retention of information, strengthening the paper's central claim.", "weaknesses": "1. While the paper evaluates performance on both image and text classification tasks, it doesn't expand into other types of tasks (e.g., regression, segmentation, or sequence-to-sequence tasks), potentially limiting the generalizability of the findings.\n\n2. The three-stage supervised learning framework has several components. An ablation study detailing the contribution of each component to the final performance would have given insights into the necessity and utility of each part.\n\n3. The paper doesn’t discuss any potential increase in computational complexity or training time introduced by the InfoR-LSF framework, which can be critical for real-world applications.\n\n4. Apart from the sensitivity analysis of coefficient α, the paper does not deeply dive into how sensitive the model is to other hyperparameters, which can be crucial for replication and understanding model robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yi7HTn9x3h", "reviewer": "Reviewer_GgyT", "summary": "This paper proposes a new paradigm of feature learning: Information Retention. The idea is to extract and keep as much *relevant* information as possible. This differs from the Information Bottleneck as such that Information Bottlenect tries to find the most compact informative features -- and this one does not focus on compactness, but keep as many relevant features. The proposed method consists of three stages that reflect the idea: (1) extract main features (2) find the most salient features -- make a copy of features with the salient ones removed (3) joint training of the features found in (1) and (2).", "questions": "1. In Weaknesses\n2. Why does the feature erasure done in sample space, and not in feature space?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written\n- The writers did a good job to soundly motivate why it makes sense to retain as much relevant feature as possible -- going slightly against the common paradigm to fins the most compact feature representation \n- Reasonable baselines and works on multiple modality\n- I like that the authors perform verification experiment (3.3); this strengthen the claim.", "weaknesses": "- One of my biggest concern is: in the abstract, the motivation was to enable better ood/distribution shift prediction. but none of the evaluation reflects this. in fact, the motivation seem to focus on low-resource settings. Does the authors focus is to generalize from learning on a small sample? If yes, this should be made clearer in the writing.\n- An algorithm table would help a lot in method clarity", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)