Regularizing Energy among Training Samples for Out-of-Distribution Generalization
Authors: Yiting Chen
Qitian Wu
Junchi Yan
Venue: ICLR 2025 Poster (2025)
Keywords: Energy based model; OOD generalization; Long-tail Recognition
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose to regularizing the energy regarding the logit on in-distribution data to improve the out-of-distribution (OOD) generalization ability of the classifier..

Abstract: The energy-based model provides a unified framework for various learning models where an energy value is assigned to each configuration of random variables based on probability. Recently, different methods have been proposed to derive an energy value out of the logits of a classifier for out-of-distribution (OOD) detection or OOD generalization. However, these methods mainly focus on the energy difference between in-distribution and OOD data samples, neglecting the energy difference among in-distribution data samples. In this paper, we show that the energy among in-distribution data also requires attention. We propose to investigate the energy difference between in-distribution data samples. Both empirically and theoretically, we show that previous methods for subpopulation shift (\emph{e.g.}, long-tail classification) such as data re-weighting and margin control apply implicit energy regularization and we provide a unified framework from the energy perspective. With the influence function, we further extend the energy regularization framework to OOD generalization scenarios where the distribution shift is more implicit compared to the long-tail recognition scenario. We conduct experiments on long-tail datasets, subpopulation shift benchmarks, and OOD generalization benchmarks to show the effectiveness of the proposed energy regularization.

Published as a conference paper at ICLR 2025
REGULARIZING ENERGY AMONG TRAINING SAMPLES FOR
GENERALIZATION
Yiting Chen', Qitian & Junchi Yan'
Sch. of Computer Science & Sch. of Artificial Intelligence, Shanghai Jiao Tong University
Eric and Wendy Schmidt Center, Broad Institute of MIT and Harvard
{s jtucyt, yanjunchi jtt juu.edu......e...u..... en, wuggttaanebracdinstttuu..e..
ABSTRACT
The energy-based model provides unified framework for various learning mod- els where an energy value is assigned to each configuration of random variables based on probability. Recently, different methods have been proposed to derive an energy value out of the logits of classifier for -i-or--distribttion (OOD) detec- tion or OOD generalization. However, these methods mainly focus on the energy difference between in in-distribution and OOD data samples, neglecting the energy difference among in-distribution data samples. In this paper, we show that the en- ergy among in-distribution data also requires attention. We propose to investigate the energy difference between in in-distribution data samples. Both empirically and theoretically, we show that previous methods for subpopulation shift (e.g., long- tail classification) such as data re-weighting and margin control apply implicit energy regularization and provide unified framework from the energy per- spective. With the influence function, we further extend the energy regularization framework OOD generalization scenarios where the distribution shift is more implicit compared to the long-tail recognition scenario. We conduct experiments on long-tail datasets, subpopulation shift benchmarks, and OOD generalization benchmarks to show the effectiveness of the proposed energy regularization.
INTRODUCTION
Energy-based models (EBMs) (LeCun et al., 2006; Ranzato et al., 2006; 2007) provide a unified theoretical framework for various learning models where energy value is assigned to each config- uration of random variables regarding its probability. Derived from the logits of a classifier, previous works show that discriminative modpel is also an energy model (Xie al., 2016; Grathwohl al., 2020) and use it for generative tasks. Inspired by this, the corresponding energy model is employed for OOD detection (Liu et al., 2020; Bitterwolf al., 2022; Wu et al., 2023). cut-of-distribution (OOD) data samples are detected by higher energy value than in-distribution(IID) data samples due to the classifier's low probability of seeing out-of-distribution data samples. Recent work (Xie et al., 2022) also minimizes the distance of energy distribution between the source domain and target domain to enhance the performance in domain adaptation. However, these previous works mainly focus on the energy difference between IID data and OOD data, neglecting the energy difference between in nn-iitributtin data samples.
In this paper, propose investigate the energy difference between in-distribution data samples (training data samples) and regularize the energy on training samples to boost the OOD generaliza- tion performance. From the energy regularization perspective,
both empirically and theoretically show that long-tail recognition methods (Wang et al., 2017; Zhou et al., 2018; 8; Liu et al., 2019; Zhong et al., 2019; He et al., 2021; Zhong al., 2021) (a special case of sub- uub-population shift (Cai et al., 2021; Koh et al., 2021)) such as reweighting data (Zhang et al., 2018; Zhao et al., 2019; Ye al., 2020; Hsich al., 2021) or controlling the classification margin (Cao et al., 2019) could be
*The work was mostly completed when Qitian Wu was Ph.D. student with SITU
i 'Correspondence author who is also affiliated with Shanghai Artificial Intelligence Laboratory. The SITU authors were partly supported by NSFC (62222607), Shanghai Municipal Science and Technology Major Project (2021 SHZDZX0102), and Seed fund of SJTU-UCI. research program.

---

Published as a conference paper at ICLR 2025
Unregalarizal Eurry
Influrnce Eueryo
Previous method
Differences between previous method and our method
E(x)
80o
100
Energy- based OOD detection
Previous works using energy to deal with OOD mainly focus on the energy difference between OOD data and in-distribution data, neglecting the energy dif- ference between in-distribution data samples themselves.
IAER
LogSummxp
Regularizul Energy
Clessilier
Data re- weighting and mar- gin control
These methods implicitly af- fect the energy value of in- distribution data. We provide a unified framework for them.
Training data didribbation
Texting data ditribation
X ~ yoNu?
100
Invariant Risk Mini-
These methods propose regular- izing the risk among training do- mains. Instead of focusing the risk. propose orthog- onal method that regularizes the energy among training domains.
mization
Figure 1: Illustration of our proposed Influence-Aware Energy Regularization (IAER). We quantify the influence of en- ergy regularization and apply regulariza- tion accordingly.
Table I: Some previous works and ours
regarded as implicit regularization on the energy value of training samples. For domain general- ization, a main branch of works focuses on invariant risk minimization (Chang et al., 2020; Creager et al., 2021; Lin et al., 2021), which regularizes the risk across different training domains to learn an invariant classifier. While the energy value is unrelated to the risk, we show that regularizing energy value across training domains could also improve the OOD generalization and is orthogonal to the methods for invariant risk minimization. We propose an energy regularization method and verify its effectiveness on tasks in long-tail classification, subpopulation shift, and domain generalization. Other empirical findings regarding the energy distribution of training data samples are provided. We summarize the contributions of this paper in the following:
Besides the extensive research for OOD detection and OOD generalization focusing on the energy difference between out-of-distributio data and in-distribution data (Liu al., 2020; Bitterwolf et al., 2022; Wu et al., 2023; Xie et al., 2022), to the best of our knowledge, we are the first to call for attention the energy difference between in-distribution data samples.
We theoretically show that data re-weighting and margin control, the two branches of methods proposed for long-tail classification (a case for subpopulation shift) could be unified as implicit energy regularization among in-distribution data samples.
We propose method namely influence aware energy regularization to regularize energy among training domains boost the model performance on OOD generalization.
RELATED WORKS
Energy Based Learning. Energy-based models (EBMs) (LeCun et al., 2006; Ranzato et al., 2006; 2007) provide unified theoretical framework for various learning models. Recent works employ the energy function defined on discriminative models for other tasks e.g. generative learning or OOD detection. Xie et al. (2016) shows that a generative random field model can be derived from a discriminative neural network. While Grathwohl et al. (2020) finds that neural classifiers are also energy- based models for joint distribution and devises hybrid model that acts as both discriminative and generative models. Liu et al. (2020) proposes to use the energy value to detect OOD samples, which has been theoretically proven (Bitterwolf et al., 2022) to be equal to training an additional binary discriminator. Recent work (Xie et al., 2022) minimizes the distance of energy distribution between the source domain and target domain to enhance the performance in domain adaptation (see Wang & Deng (2018) and the references therein). However, these previous works focus on the energy difference between in-distribution data and out-of -ii-ofi-ittiitiiion data, and we show that the energy difference between in-distribution data points could influence the generalization.
Long- Tail Recognition. Long-tue tail recognition has drawn increasing attention (Wang et al., 2017; Zhou al., 2018; Liu al., 2019; Zhong et al., 2019; He et al., 2021; Zhong et al., 2021) due
2

---

Published as a conference paper at ICLR 2025
to the pervasiveness of imbalanced data in real-world scenarios. Most methods could be divided into three categories: re-sampling the data, re-weighting the loss, and transfer learning. For re- sampling, various methods have been proposed to re-sample the dataset for more balanced data distribution (Chawla et al., 2002; Estabrooks et al., 2004; Han et al., 2005; Liu et al., 2009; Shen al., 2016; Liu et al., 2019; Wang al., 2020; Kang al., 2020; Zhang & Pfister, 2021). As for re-weighting, re-weighting methods assign different losses to different classes (Zhang et al., 2018; Zhao et al., 2019; Ye et al., 2020; Hsich al., 2021) or different data samples (Lin al., 2017; Ren et al., 2018; Shu et al., 2019) to balance the performance on each class. Specifically, LDAM (Cao et al., 2019) proposes a iustribution-awarr loss that enlarges the margin to less frequent (tail) classes.
Subpopulation Shift. Subpopulation Shift focuses on changing the proportion of some subpop- ulations (Cai et al., 2021; Koh et al., 2021), where subpopulations refer to subsets of a data do- main divided by certain attributes. A conventional setting defines subpopulations as the product of attributes and classes (Geirhos et al., 2020). In fact, long-tail recognition is a special case of subpopulation shift. Similar to long-tail classification, models tend to learn spurious features when minimizing overall loss, resulting in poor performance on minority subpopulations (DeGrave et al., 2021; Joshi et al., 2022). A wide array of methods has been developed, some focusing on scenarios where attributes are known (Gowda et al., 2021; Izmailov et al., 2022; Menon et al., 2020; Nam al., 2022; Sagawa et al., 2019; Yao et al., 2022), while others investigate cases where attributes are unknown (Creager et al., 2021; Han et al., 2022; Idrissi et al., 2022; Liu et al., 2021a).
Domain Generalization. Specifically, domain generalization (DG) (Blanchard al., 2011) aims to train model using data from single or multiple source domain that would generalize well to any out-of-distributionnOO11
target domains. Various methods have been proposed tackle the domain generalization problem including learning domain-invarian representations (Muandet et al., 2013: Li et al., 201 8b;c), augmenting the data (Zhou al., 2020; Yan et al., 2020) and applying meta-learning to domain generalization (Li et al., 201 8a; Balaji et al., 2018). Some of the methods propose regularization strategies designed based on heuristics that surpass the predictive power of an auxiliary CNN implemented as stack of 1 x convolution layers (Wang et al., 2019) or mask out the features with large gradients (Huang et al., 2020). See Wang et al. (2021); Zhou et al. (2021) for comprehensive survey.
3 ASSOCIATING METHODS IN LONG-TAIL RECOGNITION WITH TRAINING ENERGY REGULARIZATION
Various methods for long-tail recognition (Wang et al., 2017; Zhou et al., 2018; Liu et al., 2019; Zhong et al., 2019; He al., 2021: Zhong et al., 2021) (a case of subpopulation shift (Cai et al., 2021; Koh et al., 2021)) have been proposed with different motivations. In this section, we show that these different long-tail recognition methods implicitly change the training energy and that energy regularization among training data samples unifies two different branches of long- loig-tail recognition methods (reweighting (Zhang et al., 2018; Zhao et al., 2019; Ye et al., 2020; Hsieh et al., 2021) and margin control (Cao et al., 2019)).
3.1 METHODS FOR LONG-TAIL RECOGNITION ARE IMPLICIT ENERGY REGULARIZATION
For a K K-class classification problem, a parameterized classifier fo RD + R maps data point E RD to K real-valued logits where is the trainable parameter. For data point x and its corre- sponding label y. the loss for the parameter 0 is defined as L(x, y, 0). Energy-based model (LeCun et al., 2006; Grathwohl et al., 2020) E(x) RRD + IR maps each data point x to a single, non- probabilistic scalar called the energy, where the energy value could be turned to probability as:
E(x)/TT
p(x)
Je E(X)T
For classifier fe, the logits are typically converted to a normalized probability distribution with the exp(fa(x)[x)
Softmax function: Po(y|x) Σ, exp(ff(x)] where [x)) represents the logit corresponding to the y-th class. The joint distribution of data x and label could be defined as (x, 3) exp fx v/)
Z(0)
where 7(0) is unknown normalizing constant. By marginalizing out y, the unnormalized density
3

---

Published as a conference paper at ICLR 2025
model for x is (x(x) = Eg Po y/) = Ey (xp(Jo(xx)1]]
Therefore the energy at data point x
2(0)
regarding to Po could be defined as:
Ex(x) log Σ exp (fo(x)
The energy defined on classifiers is firstly introduced for generative tasks (Xie et al., 2016; Grath- wohl et al., 2020), where a classifier could be treated as an EBM and used for image generation. The energy of classifiers is also used for OOD detection (Liu et al., 2020; Bitterwolf et al., 2022: Wu et al., 2023), where unseen data samples (OOD samples) generally have higher energy.
Similar to the OOD detection scenario, the energy is lower on sub- populations with large amounts of data in the sub-population shift scenarios (Cai et al., 2021; Koh et al., 2021) (e.g. the classes having much more data samples than other classes in long-tail classification). In this section, taking long- tail recognition as a typical and clear case of sub-population shift, we show that various methods proposed to tackle the long-tail recognition problem apply implicit energy regularization.
Data resampling or re- re-weighting (Zhang et al., 2018; Zhao et al., 2019; Ye et al., 2020; Hsich et al., 2021) is one of the main branches of methods for long-tail recognition. By assigning different weights to different classes or sampling the data from the minor classes with little data more often, these methods encourage the model to reduce the loss of training data samples from the minor classes. Similar to the OOD detection scenario, the energy on more frequently trained data would be lower, which implicitly regularizes the energy on training data samples. Cao et al. (2019) also proposes margin control for better long-tail tail recognition performance. The classification margin is the difference between the logit of the ground -rrth label and the largest logit of non- non-ground-trutt labels. By controlling the classification margin be larger for data samples from minor classes, the method prevents the model from misclassifying minor classes and improves the model performance. By controlling the margin, the logits are enlarged for the data samples from minor classes, which also implicitly regularizes the energy and encourages more uniform energy distribution between the minor and major classes.
In Fig. 2, we train ResNet-32 on CIFARIO-LT and CIFARI00-LT following Cao et al. (2019) and report the average energy of each class. Generally, for the models trained with ERM (Vapnik, 1998),
the energy is lower on the classes with more data samples, indicating larger probability p(x)
(Pearson Correlation Coefficient is at -0.74 on CIFARIO-LT and -0.60 on CIFARI00-LLT) This phenomenon is because the model is trained to give lower energy (corresponding to higher p(x))
to the frequently trained data, which has been utilized to detect OOD samples (Liu et al., 2020).
On the other hand, the models trained with LDAM have more uniform energy distribution among classes (Pearson Correlation Coefficient is at -0.26 on CIFARIO-LT and 0.16 on CIFAR100-LTT It empirically shows how the margin control implicitly affects the energy among training samples. In the next section, we theoretically unify the sample re-weighting and margin control from the energy regularization perspective.
As a special case of sub- sub-population shift, the long-tail recognition shows a clear probability shift as the probability of data samples from some classes higher. Methods such as re-weighting or margin control implicitly affect the energy, pushing the p(x) predicted by the model closer to following a uniform distribution, the ground truth distribution in the test set.
3.2 UNIFYING REWEIGHTING AND MARGIN CONTROL IN ENERGY REGULARIZATION
To deal with the imbalanced distribution among different classes in long-tail recognition, many previous works resort to assigning different weights to samples from different classes (Zhang al., 2018; Zhao et al., 2019; Ye et al., 2020; Hsich et al., 2021), while other works such as LDAM (Cao et al., 2019) adjust the margin to the decision boundary for different classes. In this section, we show that energy regularization actually unifies both data re-weighting and margin control. Starting with a cross-entropy loss, for a data sample x and the ground truth label y, we have
exp(fo(x)
(xe(x, v. = log Ey exp( (x)[x') - fx(()) - Ex(
(3)
Adding an energy regularization with coefficient Ax regarding the input x, we have Lee (x, y, Bx (x(x) -o (1 -a) Fog (x)
(4)
4

---

Published as a conference paper at ICLR 2025
-75-
Average Energy
Average Energy
-3-
19-
-125
ERM
LDAM
150.
-290-
LOAM
1900
2008
<olo
5002
360
400
Sample Number
Sample Number
(a) CIFARIO-IT
(b) CIFAR100-LT
Figure 2: Average energy of each class of ResNet-32 trained on CIFARIO-LT and CIFARIOO-LT. We follow the hyper-paramees setting in Cao al. (2019) train models. Each dot corresponds to class. The model trained by ERM has biased energy distribution, where energy is generally lower for the class with more data. (The Pearson correlation coefficient is -0.74 on CIFARIO-LT and -0.60 on CIFAR 100-LT.) As for the model trained with LDAM (Cao et al., 2019), the average energy of different classes is nearly identical. (The Pearson correlation coefficient is -0.26 on CIFAR LO and 0.16 on CIFAR (00-LT)) The results empirically indicate the LDAM is actually an implicit energy regularization and levels the energy on samples from different classes
Then the gradient of the loss is:
O[ (x, y, 0) + 3x Eo Ex()
(11-Bw) (xx)))) p(u))) -1] 0
p(y'lx)) [f((x)(v]
+(1- 3x) Σ
00
00
00
(5)
When 3x # 1, we could further derive the gradient as:
ØC(x, 0) (1 (1- plulx) 1 Bx
2 (x)
(x)())
+ +(1 -1-3xx)
Σ plll y'
00
00
00
where [(x, y, 0) = Lee (X, y, Bx (x). As shown in Eq. 6, the influence of energy regulariza-
tion is twofold: adjusting the margin and reweighting data points. The weight for each data point is 1- 3x. As for the margin, the margin is defined as f= my f())) in Cao et al. (2019). In Eq. 6, the coefficient of the gradient (xx(x)] is changed from p(y) x) 1 to p(y]x) 1-3, 50
When 0 < < 1, the energy regularization enlarges the margin by pushing down the coeffi- cient of (x) and down-weights the data point with coefficient (1 3xx) When &x < 0, the
28
regularizer reduces the margin and up-weights the data with coefficient (1 -x). Therefore, we theoretically show that regularizing energy is actually combination of data reweighting and margin control unifying two different branches of methods in long-tail recognition.
REGULARIZING ENERGY FOR OOD GENERALIZATION
In Sec. 3, we show that long-tai recognition methods such LDAM (Cao et al., 2019) implic- itly regularize the energy. In this section, we want to extend the energy regularization to a more general OOD generalization scenario. the long-tail recognition scenario, the disparity of data amount between different classes is known, guiding the data re- re-weighting or margin control. Since the distribution shift in the OOD generalization scenario is more implicit, it requires a method to determine the energy regularization coefficient 3x. Note that energy regularization is orthogonal to the previous methods regularizing the risk among different domains the reasons
follows.
Motivation: The energy correlates to the probability of data sample p(x) predicted by the model. Similar to the long-tail recognition scenario, we wish the predicted probability be close to the
5

---

Published as a conference paper at ICLR 2025
actual probability in the test domain. While have way of knowing the test data distribution, the least we can do is to prevent the model from being over-confident (i.e. expect the unexpected). Note that the energy does not corresonnd to the prediction of the classifier (Remark 4.1), which the reason most previous works overlook the energy disparity between training data samples and also make the energy regularization orthogonal to previous works focusing regularizing the risk (Chang et al., 2020; Creager et al., 2021; Lin et al., 2021).
Remark 4.1. [Arbitrary Energy] Consider v(x. y) Dirain and classifier fe. For VE E R. there exists a classifier Dy that satisfy
po(y)x) (x(x) = E.
= (,(()x),
where po (yl x) and P, (y) x) is the conditional probability predicted by fo and 9n, respectively while (x,(x) (x) is the energy value of 9n data pointx. x.
4.1 DETERMINING ENERGY REGULARIZATIO COFFFICIENT VIA INFLUENCE FUNCTION
The difference between a typical OOD generalization scenario and long-tail recognition is that the distribution shift is implicit. One of the main challenges in applying energy regularization is determining the coefficient 3 in Eq. 5 as we do not know the testing data distribution. In this paper, we introduce the influence function to determine the coefficient B. Influence func- tion (Cook & Weisberg, 1982) was used to determine the influence of training data samples on the model performance (Koh Liang, 2017). Similar to that in Koh & Liang (2017), given a training set with data points Dtrain
((x1.81), (x): ... (xn, Bn)}, the optimal parame- ter for empirical risk is given by 0 def arg ming % > [(x,, v.. 0). When we add an energy regularization on a training data point (x, y) with a small e, the new optimal parameter becomes 0 (_((x.v) = arg ming ! Σx= L(x,, Vi, 0) + <F)())). Assume that the empirical risk is twice- differentiable and strietly convex w.r.t. 0, the influence function provides the influence of the energy regularization on (x, y):
Ig(x, v) def dd,,(x..) de Ie=0 -H6 Vo
-Hg"
where H1 - 4 Vjc(x; Vi, 0) is the Hessian matrix. Using the chain rule, the influence on the loss at a validation point (Xna, Yoa) is:
Ixxu Vna (x, y) def V VoL (xvas Mua, 0) ddc.e de le=0
VoVo((zesst 0) HJ V (x)).
This definition is similar to that in Koh & Liang (2017). Based on our devised influence function of energy, we propose principled method that introduces Influence Aware Energy Regularization (we refer to it as IAER), It firstly calculates the average influence of energy regularization validation set Drat = {(x]"² val y ) for each training data point. To fairly compare with previous works, take subsets of the training set as the validation set without introducing any additional data. The average influence of energy regularization on the validation set is
m
Σ Tvat(x;) = m ;=1
Toggat wpan (x;)
(10)
To reduce the loss on the validation set, we increase the energy on the data points with positive influence of energy regularization and decrease the energy on those with negative influence. There- fore, we finetune the model with energy penalties determined by the corresponding influence value. The coefficient 3x, for data x; is
Bx, -y (x:) /
(11)
Here Imaa is the maximum absolute value of influence value over the train set Dtrain and we set the hyperparameter < y < 1, by which we make sure that the energy regularization does not interfere the optimization cross-entropy.

---

Published as a conference paper at ICLR 2025
Table 2: Average testing accuracy (%) of method Imbalanced CIFAR
Dataset
Imbalanced CIFARIO
Imbalanced CIFARI00
Imbalance type
long-tailed
step
long-tailed
step
Imbalance Ratio
100
10
100
10
100
10
100
10
ERM (Cao et al., 2019)
70.36
86.61
63.30
84,27
38.32
56.59
38.55
54.70
LDAM-DRW (Cao et al., 2019)
77.16
87.62
75.36
87.42
42.04
56.67
45.36
56.84
ERM + TAER
75.83
87.02
71.33
85.62
39.60
57.59
39.12
55.10
IDAM-DRW TAER
78.37
87.72
75.89
87.70
42.81
56.67
44.61
56.9
4.2 EXPERIMENTS WITH IAER IN DIFFERENT SCENARIO
4.2.1 LONG-TAIL RECOGNITION
We first conduct experiments on the long-tail recognition scenario to test our energy regularization method. We evaluate IAER on the imbalanced version of CIFARIO, CIFAR (Cui et al., 2019) and ImageNet-LT (Liu et al., 2019) that are artificially created with class imbalance and iNaturalist 2018 (Van Horn et al., 2018), a naturally long-tailed dataset. Experiments are conducted on imbal- anced CIFAR following Cao et al. (2019) and on ImageNet-LT following Kang et al. (2020). For a fair comparison, the validation set used to calculate the influence function is sampled from the training set, and the models are not exposed to testing data during training.
1) Empirical risk minimization (ERM): with equal weight for each training data, and the model is trained to minimize the cross- entropy. 2) LDAM-DRW (Cao et al., 2019): LDAM introduces a -iber-distriuution aware margin loss, enlarging the decision while DRW applies re- weighting or re-sampling after the last learning rate decay. Since ERM is the basic training algorithm and a typical baseline while LDAM- -RWW achieves SOTA imbalanced CIFAR datasets, take these two methods as baselines.
Results on CIFAR. We evaluate IAER with ResNet-32 trained by:
CIFARIO 10 and CIFAR100 both contain 50, 000 images in training and 10, 000 images in testing with 10 and 100 classes, respectively. We construct the imbalanced version of CIFARIO 10 and CIFAR by reducing the number of images for each class. Two types of imbalance are considered: long-tailed imbalance (Cui et al., 2019) and step imbalance (Buda et al., 2018). For long-tailed imbalance, the number of data points follows an exponential decay across different classes. For step imbalance, data points in half of the classes are reduced to the same number while the number of data points in the other classes remains the same.
As shown in Table 2, our method could effectively boost the testing performance after only 5 epochs. The more imbalanced the IAER is, the more effective it is. Notably, IAER could greatly improve the ERM pre-trained model. For instance, IAER reduces the testing error of the ERM pre-trained model for 5.47% (from 29.64% to 24.17%) on long-tailed CIFARIO 10 with the imbalance ratio at 100. For CIFAR 100, IAER also improves the testing performance for the ERM pre-trained model and improves the LDAM-DRW pre-trained model on boillt CIFAR 100 with the imbalance ratio at 100 and step-imbalancee imbalanced CIFAR 100 brought by our TAER is much smaller than that of imbalanced CIFAR 10. We conjecture that the calculated influence of energy regularization on our sampled validation set for CIFARIO0 is less accurate since the number of images per class in CIFARIO0 is much smaller than that of CIFARIO egg. only 5 images for the least frequent class when imbalance ratio is 100.
CIFAR100 with the imbalance ratio at 10. However, the improvement in
Results on ImageNet-LT And iNaturalist 2018. We evaluate IAER with ResNeXt-50 (Xie al., 2017) pre-trained by the techniques and protocols proposed in Kang et al. (2020) where each model divided into two parts: backbone and linear classifier. The protocol includes (1) Classifier Re- training (cRT): employ the backbone trained with ERM and retrain the linear classifier with the class balance sampling method. (2) Learnable Weight Scaling (LWS): Rescale the weight of the classifier for each class by rescaling factor learned with the class balance sampling method as in cRT. ImageNet-LT (Liu et al., 2019) is artificially truncated from ImageNet (Deng et al., 2009), where the label distribution follows long-tailed distribution. It has 1000 classes, and the number of images per class ranges from 1280 to 5. iNaturalist 2018 (Van Horn et al., 2018) is a real-world, long-tailed dataset with 8142 classes. We follow Liu et al. (2019) and report the testing accuracy on three kinds of class sets: Many-shot (over 100 images), Medium-sher (20 ~ 100 images) and Few-shot (less than 20 images). The testing accuracy on all classes is denoted All.
7

---

Published as a conference paper at ICLR 2025
Table 3: Experiments ImageNet-I.T and iNaturalist. The validation set for IAER is composed of
images in the train set.
Dataset and model
ResNeXt-50 ImageNet-LT
ResNet-115 iNaturalist
Method
Many
Median
Few
All
Many
Median
Few
cRT (Kang et al., 2020)
61.8
46.2
27.4
49.6
75.9
71.9
69.1
71.2
cRT |AER|
61.0
45.6
29.1
49.3
76.1
71.6
69.5
71.2
cRT IAER|M IAER|Mcdiaal Mcdian]
58.5
48.7
26.0
49.4
75.8
72.3
68.0
71.0
cRT Many|
62.7
44.5
26.9
49.1
77.8
69.9
66.5
69.4
LWS (Kang et al., 2020)
60.2
47.2
30.3
49.9
74.3
72.4
71.2
72.1
LWS [AER[Few]
60.1
47.1
32.1
50.1
74.4
72.4
71.6
72.3
LWS IAER[Median]
58.1
49.0
30.3
50.0
74.5
72.8
70.9
72.2
LWS + [AER[Many]
61.5
45.5
30.1
49.6
74.9
72.6
71.0
72.2
Table 4: Experiment results for subpopulation shift conducted based on SubpopBench (Yang et al., 2023) on CMNIST (Arjovsky et al., 2019), MetaShift cats us. dogs (Liang & Zou, 2022), NICO+ (Zhang et al., 2023), Waterbirds (Wah et al., 2011) and CivilComments (Borkan et al.,
2019).
Dataset and Method Metric
ERM
ERM
IAER
Mean
Worst
Mcan
Worst
CMNIST
77.8
4.6
78.1
14.3
MetaShift
90.4
66.2
90.6
67.7
NICO+
82.0
30.0
82.8
33.3
Note that the minimum number of data points per class in the training set of iNaturalist is 2, which makes the possible class-balanced subset of the training set extremely small. Therefore, for ImageNet-LT and iNaturalist, in the training set as the validation set, respectively. We employ the backbone provided by Kang al. (2020) and finetune or retrain the classifier. For ImageNet-LT, we calculate the influence of energy regularization on the ResNeXt-50 pre-trained for 90 epochs and retrain the classifiers with IAER for 10 epochs. For iNaturalist 2018, we calculate the influence of energy regularization of the ResNet- 152 pretrained for 200 epochs. The classifiers on the iNaturalist are retrained for 30 epochs with energy regularization. More details are in Appendix A.
take images of Many-shot, Medium-shot, and Few-shot classes
As shown in Table 3, [AER[Few] means that the validation set used for calculating influence is composed of images of few-shol classes the training set while IAER[Median] and [AER[Man]] means the validation set is composed by median-shot classes and many-shot classes respectively. For ImageNet-LT and iNaturalist 2018, we could observe that the accuracy on the classes used to calculate the influence function is boosted e.g. IAER[Few] boosts the accuracy of the classifier on few-shot classes while [AER[Median] boosts the accuracy of the classifier on median-shot classes. LWS combined with IAER[Few] could improve the accuracy of the whole testing set.
4.2.2 RESULTS FOR SUBPOPULATION SHIFT
We follow SubpopBench (Yang et al., 2023) to conduct experiments for subpopulation shift. Sub- populations are subgroups of data domain divided based on certain features, and subpopula- tion shift is type of distribution shift characterized by changes in the proportion of some sub- populations. The long-tai tail dataset could be seen as a special case of subpopulation shift. We con- duct our experiments et al., 2019), MetaShift cats vs. dogs (Liang & Zou, 2022), NICO+ (Zhang et al., 2023). For fair comparison,
widely used datasets in SubpopBench, including ColoredMNIST (Arjovsky
calculate the influence function. For the model training, we follow the setting in SubpopBench. As shown in Table 4, both the mean accuracy and the worst accuracy are improved with our IAER.
the training set as the proxy validation
4.2.3 RESULTS FOR DOMAIN GENERALIIZATION
We follow DomainBed (Gulrajani & Lopez-Paz, 2021) to conduct experiments for domain gener- alization. DomainBed is a testbed for domain generalization. As Gulrajani & Lopez-Paz (2021) shows that ERM outperforms SOTAs by average performance common benchmarks evaluated
8

---

Published as a conference paper at ICLR 2025
Table 5: Experiment results for domain generalization conducted CMNIST (Arjovsky al,, 2019), PACS (Li et al., 2017) and VLCS (Fang et al., 2013). means we use the results from Domainbed (Gulrajani Lopez-Paz, 2021))
Method
CMNIST
PACS
VLCS
ERM (Vapnik, 1998)
51.5
0.1
77.4 + 0.2
IRM (Arjovsky al., 2019)*
52.0
83.5
78.5 + 0.5
GroupDRO (Sagawa et al., 2019)*
52.0
8.84
76.7
MLDG (Li et al., 2
5.1
8.0
77.2
CORAL (Sun & Saenko, 2016)
5.1
8.21
78.8 0.6
SagNet (Nam et al., 2021)
51.7 1.77000
86.3
77.8 = 0.5
ERM + LAER
0.1
86.6 0.1
78.5 0.2
Average Influence Energy
-0
0,00005
00000
Influence of Energy
00000
rbect
-0.0013
Lhose
-0 0.0000
cbend
chess
-0.0023
(los7
-a1e005-
Sample Each
Train Loss
Figure 3: The average influence en-
Figure 4: Influence of energy regulariza- tion and loss of each training data point (a dot in the plot) in long-tailed CIFARIO for ResNet-32 trained with ERM.
ergy regularization dot represents a class, and the x-axis corre- sponds to the number of data samples.
CIFAR1O-LT. Each
in a consistent and realistic setting, we combine our method with ERM and take the algorithms implemented in DomainBed as baselines. Experiments are performed on benchmarks ColoredM- NIST (Arjovsky et al., 2019), PACS (Li at. 2017) and VL.CS (Fang et al., 2013).
As mentioned in Sec. 4.1, we take the training set as the validation set to calculate the influence function for a fair comparison. We follow the setting in DomainBed (Gulrajani Lopez-Paz, 2021) to train the model. We demonstrate the results using training domain validation as model selection criteria, which use a validation set sampled from the training domain for model selection. For each algorithm and testing domain, we conduct random search of 5 trails. For more experimental details, see Appendix A. As shown in Table 5, our IAER could improve the accuracy on the test domain without the test domain information. This implies that regularizing energy the training domains helps generalize the training domains to the testing domain. More results are in Appendix B.
OTHER EMPIRICAL RESULTS
AVERAGE INFLUENCE OF ENERGY REGULARIZATION ON CIFARIO-LT
We plot the average influence of energy regularization of each class on CIFARIO-LT. As shown in Fig. 3, the average influence of energy regularization of different classes is positively correlated to the number of data points of the corresponding class (The Pearson's R 0.70 for long-tailed CIFAR10 with imbalance ratio at 10). The lower the influence of the energy regularization, the lower the testing loss of the classifier would be after adding a positive energy regularization. indicates that pushing down the energy value of data points of less frequent class and pulling up the energy value of data points of more frequent class would boost the testing performance i.e. pull up the predieted p(x) for less frequent class and push down the predicted p(x) for more frequent class. Since the probability density p(x) of data points of less frequent class much lower and the p(x) of data points of more frequent class is much higher in the imbalanced training set compared the

---

Published as a conference paper at ICLR 2025
Table 6:
Time required to approximate the influence function on different datasets.
Model ResNet-32
Dataset
iteration
repeat times
time used
CIFARTO TO
5000
10
718.20s
ResNeXt-50
ImageNet-LT
1000
10
8492.92s
ResNet-152
iNaturalist 2018
1000
10
9701.74s
testing set, it shows that pushing the predicted p(x) closer to the real p(x) of the testing set would boost the testing performance. According Sec. 3.2, our regularization method generally enlarges the margin and down-weight for the less frequent classes. On the other hand, it reduces the margin and up-weights for the more frequent classes.
5.2 INFLUENCE OF ENERGY REGULARIZATION ON DATA SAMPLES WITH DIFFERENT LOss
To investigate the relationship between the influence of energy regularization and the loss on each training data point, we provide a scatter figure as in Fig. 4. Note that the influence of energy regu- larization on the data points of similar training loss ranges from positive to negative. The influence of energy regularization is not related to the training loss (Pearson's ' R is -0.04). This indicates that one could not predict the influence of energy regularization on the data point based on train- ing loss. However, the range of the influence of energy regularization expands as the training loss decreases. Therefore, data points where energy regularization has a large influence are generally well-classifedd data points with low loss. As pointed out in Remark 4.1, the energy value is unstable during the training. we conjecture that the un- un-reuularized energy value of well-classffied data points is one of the possible reasons for the overfitring of the classifier.
5.3 TIME COMPLEXITY ANALYSIS FOR THE APPROXIMATION OF INFLUENCE FUNCTION
The main overhead of the proposed IAER is calculating the influence function of energy regular- ization. We calculate the influence function with stochastic estimation (Cook & Weisberg, 1982) following Koh & Liang (2017). We implement the calculation of the influence function based on the Python package for calculating the influence function (Lo & Bae, 2022). For imbalanced CIFARIO and imbalanced CIFAR 100, the influence function is calculated with stochastic estimation for 5000 iteration and averaged over 10 trails. For ImageNet-LT and iNaturalist, we calculate the influence function only on the classifier, and the influence function is calculated with stochastic estimation for 1000 iteration and averaged over 10 trails.
Since approximate the influence function using stochastic approximation, the time used cal- culate the influence function is determined by the choice of hyperparameters. We tested the time cost for calculating the influence function for ResNet-32 on CIFARIO with Intel(R) Xeon(R) CPU ES-2678 v3 @ 2.50GHz and one GeForce RTX 2080Ti for 5000 iteration and averaged ten times. Approximating the influence function for a 5000 iteration takes 718.20s on average. As shown in Table 6, we report the time used to approximate the influence function on different datasets for different models with the same hardware settings.
CONCLUSION
In this paper, we propose to regularize the energy among training data samples. We first show that various methods for long-tail recognition implicitly apply energy regularization, which pushes the energy distribution close to the test distribution. We further propose an influence-aware energy regularization for various OOD generalization scenarios, such as subpopulation shift and domain generalization. The main limitation comes from the use of the influence function. First of all, it requires a validation set. In this paper, we use a subset of the training set as the validation set. Ex- perimental results show that the choice of the validation set largely affects the performance of IAER. Secondly, as defined for convex loss functions, the influence function may not reflect the actual in- fluence for neural networks. The approximation of the influence function is also bottleneck. which is hard to converge and requires high computational cost. We hope the idea of regularizing energy on training data samples will innovate future works.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. CoRR, abs/1907.02899,, 2019.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gener- alization using meta regulaaittaa.....on.... In NeurlPS, pp. 1006-1016, 2018.
Julian Bitterwolf, Alexander Meinke, Maximilian Augustin, and Matthias Hein. Breaking down out-of off-iiitributiinn detection: Many methods based on OOD training data estimate a combination of the same core quantities. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 2041-2074. PMLR, 2022.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In NIPS, pp. 2178-2186, 2011.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion pro- ceedings of the 2019 world wide web conference, pp. 491-500, 2019.
Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-259, 2018.
Tianle Cai, Ruiqi Gao, Jason Lee, and Qi Lei. A theory of label propagation for subpopulation shift. In International Conference on Machine Learning, pp. 1170-1182. PMLR, 2021.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, and Tengyu Ma. Learning imbalanced margin loss. In NeurlPS, pp. 1576, 2019.
datasets with
Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. Invariant rationalization. In International Conference on Machine Learning, pp. 1448-1458. PMLR, 2020.
Nitesh Chawla, Kevin W. Bowyer, Lawrence Hall, and W. Philip Kegelmeyer. SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res., 16:321-357, 2002.
R Dennis Cook and Sanford Weisberg. Residuals and influence in regression. New York: Chapman and Hall, 1982.
Elliot Creager, Jörn- Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference Machine Learning, pp. 1189-2200. PMLR, 2021.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge J. Belongie. Class-balanced loss based on effective number of samples. In CVPR, pp. 9268-9277. Computer Vision Foundation/ / IEEE, 2019.
Alex J DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence, 317)6610-61, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR. pp. 248 255.. IEEE Computer Society, 2009.
Andrew Estabrooks, Taeho Jo, and Nathalic Japkowicz. A multiple resampling method for learning from imbalanced data sets. Comput. Intell., 20(1):18-36, 2004.
Chen Fang, Ye Xu, and Daniel N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In ICCV, pp. 1657-1664. IEEE Computer Society, 2013.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-61) 663, 2020.
Sindhu CM Gowda, Shalmali Joshi, Haoran Zhang, and Marzyeh Ghassemi. Pulling up by the causal bootstraps: Causal data augmentation for pre-training debiasing. In Proceedings of the 30th ACM International Conference Information Knowledge Management, pp. 606-616, 2021.
11

---

Published as a conference paper at ICLR 2025
Will Grathwohl, Kuan-Chieh Wang, Jömn- Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In ICLR. OpenReview.net, 2020.
Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization, In ICLR. OpenRe- netw 2021.
Hui Han, Wenyuan Wang, and Binghuan Mao. Borderline-smote: A new over-sampling method in imbalanced data sets learning. In ICIC (I), volume 3644 of Lecture Notes in Computer Science, pp. 878-887. Springer, 2005.
Zongbo Han. Zhipeng Liang, Fan Yang, Liu Liu, Lanqing Li, Yatao Bian, Peilin Zhao, Bingzhe Wu, Changqing Zhang, and Jianhua Yao. Umix: Improving importance weighting for subpopula- tion shift via uncertainty-aware mixup. Advances Neural Information Processing Systems, 35:
97704-37718, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp-
770-778, 2016.
Yin- Yin He, Jianxin Wu, and Xiu-Shen Wei. Distilling virtual examples for long-tailed recognition. In ICCV, pp. 235-244. IEEE, 2021.
Ting-I Hsieh, Esther Robb, Hwann- Tzonn Chen, and Jia-Bin Huang. Droploss for long-tail instance segmentation. In AAAI, pp. 1549-1557, AAAI Press, 2021.
Zeyi Huang, Haohan Wang, Eric P. Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In ECCV (2), volume 12347 of Lecture Notes in Computer Science, pp. 124-140. Springer, 2020.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-gros const-rroppaaccuracy In Conference Causal Learning and Reasoning, pp. 336-351. PMLR, 2022.
Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew Wilson. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems, 35: 38516-38532, 2022.
Nitish Joshi, Xiang Pan, and He He. Are all spurious features in natural language alike? an analysis through causal lens. arXiv preprint arXiv:2210.14011, 2022.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yan- nis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR. OpenReview.net, 2020.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 1885-1894. PMLR, 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wile distribution shifts. In International Conference on Machine Learning. pp. 5637-5664. PMLR, 2021.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on -bsed learning. Predicting structured data, 10)) 2006.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In ICCV, pp. 5543-5551. IEEE Computer Society, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta- learning for domain generalization. In AAAI, pp. 3490-3497. AAAI Press, 2018a.
Haoliang Li, Sinno Jialin Pan, Shigi Wang, and Alex C. Kot. Domain generalization with adver- sarial feature learning. In CVPR, pp. 5400-5409. Computer Vision Foundation JEEE Computer Society, 2018b
12

---

Published as a conference paper at ICLR 2025
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV(15), volume 11219 of Lecture Notes in Computer Science, pp. 647-663. Springer, 2018c.
Weixin Liang and James Zou, Metashift: A dataset of datasets for evaluating contextual distribution shifts and training conflicts. arXiv preprini arXiv:2202.065223 2022.
Tsung-Y Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. ICCV, pp. 2999-3001 3007. IEEE Computer Society, 2017.
Yong Lin, Qing Lian, and Tong Zhang. An empirical study of invariant risk minimization on deep models. In ICML 2021 Workshop Uncertainty and Robustness in Deep Learning, volume 1, pp. 7, 2021.
Evan Z. Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021a.
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distributtin detection. In NeurlPS, 2020.
Xu-Ying Liu, Jianxin Wu, and Zhi-Hua Zhou. Exploratory undersampling for class-imbalance learn- ing. IEEE Trans. Syst. Man Cybern. Part B, 39(2):339-550 2009.
Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He. Influence selection for active learning. In ICCV, pp. 9254 9263. IEEE, 2021b.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large- scale long-tailed recognition in an open world. In CVPR, pp. 2537-2546. Computer Vision Foun- dation IEEE, 2019.
Alston Lo and Juhan Bae. torch-influence, June 2022. URL htttps://gtthii com/
Aditya Krishna Menon, Ankit Singh Rawat, and Sanjiv Kumar. Overparameterriation and worst- case generalisation: friend or foe? In International Conference on Learning Representations, 2020.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICML (!), volume 28 of JMLR Workshop and Conference Proceedings, pp. 10-18. JMLR.org, 2013.
Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing do- main gap by reducing style bias. In CVPR, pp. 8690-8699 Computer Vision Foundation / IEEE, 2021.
Junhyun Nam, Jaehyung Kim, Jacho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. arXiv preprint arXiv:2204.02070, 2022.
Mare' Aurelio Ranzato, Christopher S. Poultney, Sumit Chopra, and Yann LeCun. Efficient learning of sparse representations with an energy-based model. In NIPS, pp. 1137-1144. MIT Press, 2006.
Marc' Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy-based framework for unsupervised learning. In ATSTATS, volume of JMLR Proceedings, pp. 371-379. JMLR.org, 2007.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. ICML., volume 80 of Proceedings of Machine Learning Research, pp- 4331-4340. PMLR. 2018.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. CoRR, abs/191 1.08731, 2019.
13

---

Published as a conference paper at ICLR 2025
Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In ECCV (7), volume 9911 Lecture Notes in Computer Science, pp. 467-4872 Springer, 2016.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight- net: Learning an explicit mapping for sample weighting. In NeurIPS, pp. 1917-1928, 2019.
Baochen Sun and Kate Saenko. Deep CORAL: correlation alignment for deep domain adaptation. In ECCV Workshops (3), volume 9915 of Lecture Notes in Computer Science, pp. 443-450, 2016.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018.
Vladimir Vapnik. Statistical learning theory. Wiley, 1998.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech- uusd birds-200-2011 dataset. 2011.
Haohan Wang, Zexue He, Zachary C. Lipton, and Eric P. Xing. Learning robust representations by projecting superficial statistics out. In ICLR. OpenReview.net, 2019.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, and Tao Qin. Generalizing to unseen domains: survey on domain generalization. In IJCAI, pp. 4627-4635, ijcai.org, 2021.
Mei Wang and Weihong Deng. Deep visual domain adaptation: survey. Neurocomputing, 312:
135-153, 2018
Tao Wang, Yu Li. Bingyi Kang. Jumnan Li, Jun Hao Liew, Sheng Tang, Steven C. H. Hoi, and Jiashi Feng. The devil is in classification: A simple framework for long-tail instance segmentation. In SCCV((4)) volume 12359 of Lecture Notes Computer Science, pp. 728-744. Springer, 2020.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In NIPS, pp- 7029-7039, 2017.
Qitian Wu, Yiting Chen, Chenxiao Yang, and Junchi Yan. Energy-based out-of-distribttion detection for graph neural networks. In The Eleventh International Conference on Learning Representa- fions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, Xinjing Cheng, and Guoren Wang. Active learning for domain adaptation: An energy-based approach. In AAAI, pp. 8708-8716. AAAI Press, 2022.
Jianwen Xie, Yang Lu, Song- Chun Zhu, and Ying Nian Wu. A theory of generative convnet. In JMLR.org,
ICML, volume 48 of JMLR Workshop and Conference Proceedings, pp. 2016.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, pp. 5987-5995. IEEE Computer Society, 2017.
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. CoRR, 0bs/2001.00677. 2020.
Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzych Ghassemi. Change is hard: A closer look at subpopulation shift. In International Conference on Machine Learning, 2023.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James 7ou, and Chelsea Finn. Im- proving out-of-distribution robustness via selective augmentation. In International Conference on Machine Learning, pp. 25407-25437. PMLR, 2022.
Han-Jia Ye, Hong- You Chen, De-Chuan Zhan, and Wei-Lum Chao. Identifying and compensating for feature deviation in imbalanced deep learning. CoRR, fass/200100333 2020.
14

---

Published as a conference paper at ICLR 2025
Xingxuan Zhang. Yue He, Renzhe Xu, Han Yu, Zheyan Shen, and Peng Cui. Nico++: Towards better benchmarking for domain generalization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16036-16047, 2023.
Yifan Zhang, Peilin Zhao, Jiezhang Cao, Wenye Ma, Junzhou Huang, Qingyao Wu, and Mingkui Tan. Online adaptive asymmetric active learning for budgeted imbalanced data. In KDD, pp. 2768 2777. ACM, 2018.
Zizhao Zhang and Tomas Pfister. Learning fast sample re- re-weighting without reward data. In ICCV, pp. 705-714. IEEE, 2021.
Peilin Zhao, Yifan Zhang, Min Wu, Steven C. H. Hoi, Mingkui Tan, and Junzhou Huang. Adaptive cost-sensitive online classification. IEEE Trans. Knowl. Data Eng., 31(2):214-228, 2019.
Yaoyao Zhong, Weihong Deng, Mei Wang. Jiani Hu, Jianteng Peng, Xunqiang Tao, and Yaohai Huang. Unequal-training for deep face recognition with long-tailed noisy data. In CVPR, pp. 7812-7821. Computer Vision Foundation IEEE, 2019.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recog- nition. CVPR, pp. 16489-16498. Computer Vision Foundation IEFE, 2021.
Bolei Zhou, Ágata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell., 40(6): 1452-1464, 2018.
Kaiyang Zhou, Yongxin Yang, Timothy M. Hospedales, and Tao Xiang. Learning generate novel domains for domain generalization. In ECCV (16), volume 12361 of Lecture Notes in Computer Science, pp. 561-578. Springer. 2020.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. CoRR, abs/2103.02503, 2021.
15

---

Published as a conference paper at ICLR 2025
A EXPERIMENT DETAILS
A.1 DETAILS FOR THE CALCULATION OF INFLUENCE FUNCTION
The calculation of the influence function requires a validation set. For imbalanced CIFARIO and CIFARI00, we sample data points of each class from the class-imbalanced training set to compose the validation set. The number of data points per class is determined by the minimum number of data points per class in the training set. For ImageNet-LT, since it has a validation split, we use the val split to calculate the influence function. For iNaturalist, since the minimum number of data points per class is APPall (2 images per class), we take the few-shot classes of the training set as the validation set.
We calculate the influence function with stochastic estimation (Cook & Weisberg, 1982) following Koh & Liang (2017). We implement the calculation of the influence function based on the Python package for calculating the influence function (Lo & Bae, 2022). For imbalanced CIFARIO and imbalanced CIFAR100, the influence function is calculated with stochastic estimation for 5000 it- eration and averaged over 10 trails. For ImageNet-LT and iNaturalist, we calculate the influence function only on the classifier, and the influence function is calculated with stochastic estimation for 1000 iteration and averaged over 10 trails.
A.2 DETAILS FOR THE EXPERIMENTS ON IMBALANCED DATASET
We follow the setting in Cao et al. (2019) to train ResNet-32 on the imbalanced CIFAR dataset and report the performance of the model at the final epoch. The model is trained for 200 epochs with SGD optimizer where the learning rate is at 0.1, momentum at 0.9, and weight decay at 2e - 4. The learning rate decayed with factor 0.01 at 160-th epoch and 180-th epoch. For IAER, we finetune the model for epochs with batch size at 128 and learning rate at le- 4.
For ImageNet-LT and iNaturalist 2018, we employ the pre-trained model provided Kang et al. (2020) and follow the setting in it to finetune the ResNeXt-50 (Xie et al., 2017) on ImgaeNet-LTT and the ResNet- 152 (He et al., 2016) on iNaturalist 2018. For ImageNet-LL, the classifier is finetuned for 10 epochs with batch size at 512 and learning rate 0.2. For iNaturalist, the classifier is finetuned for 30 epochs with batch size at 512 and learning rate at 0.2.
The y in Eq. 9 is searched in f0.1, 0.5, 1, 10) for CIFAR-LT and set to be 0.5 for ImageNet-LT and iNaturalist2018. When the absolute value of energy regularization is bigger than the cross-entropy Leo (xx"" .ff".)
loss the 7 is to be| Fx((x") ). (x) 3/T/
A.3 DETAILS FOR THE EXPERIMENTS FOR DOMAIN GENERALIZATION
We follow the setting in Gulrajani & Lopez-Paz Paz (2021) conduct experiments and use the training domain validation set to calculate the influence function. The y in Eq. 9 is set to be 0.1. We employ the same training settings and hyperparameters implemented in Gulrajani & Lopez-Paz (2021).
A.4 COMPARISON BETWEEN THE INFLUENCE OF THE CROSS-ENTROPY AND THE INFLUENCE OF THE ENERGY.
The influence function of the cross-entropy has been widely used in data valuation (Koh & Liang, 2017) and active learning (Liu et al., 2021b). As shown in Fig. 5, the influence of the cross-entropy and the influence of the energy do not correlate with each other. The influence function of the cross-entropy as in previous works (Koh & Liang, 2017; Liu et al., 2021b) evaluates the influence of reweighting the data points. As for the influence function of the energy, we focus the influence of the energy regularization, which acts as reweighting and margin control. Therefore there are fundamental differences between the influence of the cross-entropy and that of the energy.
16

---

0.0200 Published as a conference paper at ICLR 2025
Influence function the cross-entrope
040000
000012
000000
-0.00005
-0.00010
00020-0 00015-0 10000000000000000000000 00000000000000000000000 Influence functioon of the energy
Figure 5: Relation between the influence the cross-entropy and that of the energy. The influence is calculated for the ResNet-32 trained on CIFARIO where each point represents data sample.
11-3
Approximated Influence of Energy
0.0075
Approximated Influence Energy
0.0000
0.0030
-0,0075
-0,0075
-0.0300
-05
0.026
0.022
0.000
0.004
0.0004
0,0022
0.0000
0.0022
0.0004
0,0008
0,0025
(a) ERM pre-trained model (Pearson's 's = 0.9154) (b) LDAM pre-trained model (Pearson's 0.9037) Figure 6: The positive relation between the calculated influence function and the actual change in testing loss on pre-trained CNN. We plot the top 20 most influential data points.
B ADDITIONAL EXPERIMENT RESULTS
B. ADDITIONAL RESULTS TO VALIDATE THE INFLUENCE FUNCTION OF ENERGY REGULARIZATION
In addition to Fig. 6, where we calculate the average influence over the whole testing set, we ar- bitrarily pick a wrongly classified data point and calculate its influence function following Koh & Liang (2017).
We calculate the influence function for the ResNet-32 trained with ERM on the long-tail CIFARIO and long-tail CIFAR100, where the imbalance ratio is set to be 100. We plot the influence energy regularization and the actual change in testing loss after finetuning the model with energy regular- ization for 50 epochs on 100 most influential data point. The calculated influence function has a positive relation the actual change in loss (Pearson's R 0.7875 long-tail CIFAR10 and is 0.5744 on long-tail CIFAR100)
B.2 ENERGY DISTRIBUTION SHIFTS DURING TRAINING
We plot the energy distribution of the training set during the training ResNet-32 by ERM on the long-tailed CIFARI0 10 with an imbalance ratio of 100. As shown in Fig. 7, the energy distribution keeps changing during the training even though the training loss is stable e.g. from 100-th epoch to 150-th epoch. We further plot the distribution at 160, 170, 180, 190, and 200 epochs after the learning rate has decayed. As shown in Fig. 7(b), the energy distribution of the training set still changes when the learning rate is decayed and the model converged.
17

---

Published as a conference paper at ICLR 2025
epochs
spoct
cpocls
Rungy
density of data points
epochs
0.325
0.300
why
Ma
Nal
0.075
0.050
125
150
0.025
0.000
Energy
Figure 7: The energy distribution of ResNet-32 trained on the long-tailed CIFARIO. (a): the energy distribution of the training set on different epochs during the training (b): the energy distribution of the training set at 160, 170, 180, 190, and 200 epoch.
Model ResNett32
Dataset CIFARIO
iteration
repeat times
time used
5000
10
718.200
ResNeXt-50 ResNet-152
ImageNet-LT
1000
10
8492.92s
iNaturalist 2018
1000
10
9701.74s
Table 7: models.
Time required to approximate the influence function on different datasets for different
Method
Testing accuracy different test domains( %)
Colored MNIST
+80%
+90%
-90%
ERM
72.3
0.0
72.0 0.1
10.1 0.1
ERM +IAER
73.8 + 0.1
71.5 + 0.1
10.6 + 0.1
PACS
A
C
P
ERM
85.6 0.1
79.7 + 0.2
98.9 0.1.
78.0 0.I
ERM IAER
84.2 + 0.1
84.8 + 0.1
97.3 + 0.1
80.2 = 0.1
Table The detailed results for domain generalization where we report the testing accuracy($5 (%) for different test domains.
B.3 TIME COMPLEXITY ANALYSTS
The main overhead of the proposed IAER is calculating the influence function of energy regulariza- tion. Since we approximate the influence function using stochastic approximation, the time used to calculate the influence function is determined by the choice of hyperparameters, We tested the time cost for calculating the influence function for ResNet-32 on CIFARIO with Intel(R) Xeon(R) CPU ES-2678 v3 @ 2.50GHz and one GeForce RTX 2080Ti for 5000 iteration and averaged Approximating the influence function for a 5000 iteration takes 718.20 20s on average. As shown in Table B.3, we report the time used approximate the influence function on different datasets for different models with the same hardware settings.
times.
B.4 DETAILED RESULTS FOR DOMAIN GENERALIZATION
We report the detailed results of our experiments for domain generalization, as shown Table
B.5 PROOF FOR REMARK 4.1
Proof. For the classifier fo DDD - IR assume the energy on data point x E RD Ex For 34 E R. classifier 9n RDD RK could be defined that satisfies:
18

---

Published as a conference paper at ICLR 2025
vi E f1,2, , ,),,
g,(x) = fo(x) - & + Ex(x).
(12)
By adding certain value Eg Ex(x) (x) prediction of so while the energy value of On is changed to E. For the predicted Pr (y/xx) we have:
each logit [i)) the prediction of On is the same as the
Pr(U/x)= exp [S)((())) + Eg (x)]
Σ, exp [o (x) (x) - & + [x(
exp (x)
(13)
Σ, exp [Sa (x) [[]]
(y(xx))
For the energy (x) on the Iy. we have:
(x)xx)
log Σ exp 19,, (x)[i]]
=-log log E exp [x) E + Es(x))
log (xxp|Eo(x) -E] Σ cxp [fo (x)
(14)
= E -x( log Σ exp [x)]
E Ex(x) + Ex
E.
19


Reviews:
[{"review_id": "fv2qWPHvVF", "reviewer": "Reviewer_GFVu", "summary": "This paper proposes to explore the energy score between ID samples. It also theorectically suggests that previous methods for long-tail classification implicitly apply energy regularization. Some experiments are conducted to show the effectivness of the proposed engergy-based regularization.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and well-structured.\n2. Experiments are extensive to show the effectiveness of the method.\n3. Theorectical analysis makes the paper more solid.", "weaknesses": "1. It seems that the proposed method does not perform consistently better than baseline methods and relatively unstable.\n\n2. The designed influence function requires a large amount of time and computational recouces for approximating $\\beta$.\n\n3. In Fig 3, it can be observed that class 7 is a outlier. What cause this class significantly deviate the trend?\n\n4. Some tables use testing error, while some use accuracy. It would be more readable to make them consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZKXjqyo9C1", "reviewer": "Reviewer_16Hz", "summary": "This submission presents a unified framework for energy-based models applied to both in-distribution and out-of-distribution (OOD) data. While prior methods in OOD detection focus on energy differences between in-distribution and OOD samples, this work highlights the importance of examining energy differences within in-distribution samples. The submission proposes investigating these differences, showing that techniques commonly used for subpopulation shifts—such as data re-weighting and margin control in long-tail classification—implicitly apply energy regularization. This framework is further extended to address more implicit distribution shifts, as seen in OOD generalization scenarios, using influence functions to support the approach. Experimental results on long-tail, subpopulation shift, and OOD generalization benchmarks demonstrate the effectiveness of this energy regularization method. The source code will be publicly available.", "questions": "Please clarify the experimental analysis and Remark 4.1. Please see the above for more details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ To my knowledge, this work is the early work that focuses on the energy of in-distribution data and use it as a regularization for learning models. \n+ The Figure 2 well illustrates the motivation of this submision: the existing methods on long-tail problem can be reviewed as a kind of energy regularization to learn models. \n+ The writing is clear. The related work, the derivation of unified view are also clear. \n+ The expeiments domonstrate that proposed IAER can bring some improvements over baselines.", "weaknesses": "- The main concern lies with the experimental settings. Although the proposed IAER method shows some improvement over the standard baseline (ERM), it does not achieve competitive results compared to state-of-the-art methods. Specifically, Table 2 only includes comparisons with two works from 2019. Why are more recent works not included for comparison? This same question applies to Tables 3, 4, and 5. I would except more recent work to be included and disucssed. Just name a few: 1) Gradient Reweighting: Towards Imbalanced Class-Incremental Learning  2) Dynamic residual classifier for class incremental learning, 3) Towards principled disentanglement for domain generalization. 4) Flatness-aware minimization for domain generalization\n\n\n- Lines 270–282 lack clarity. For instance, the statement “The least we can do is to prevent the model from being over-confident… note that the energy does not correspond to the prediction of the classifier (Remark 4.1), which is the reason most previous works overlook the energy disparity” requires further clarification: 1) yes, we usually do not have knowledge of test distribution. But how to prevent it from being over-confident? 2) I am not clear why Remark 4.1 explains why most previous works overlook the energy disparity between training data samples. 3) Additionally, the main point of Remark 4.1 is unclear. Could you clarify its purpose and significance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MpbIeDrdvP", "reviewer": "Reviewer_f4ew", "summary": "The paper addresses the issue of OOD generalization in machine learning models. It proposes a novel approach to regularize the energy among ID data samples, which is crucial for improving OOD generalization performance. The authors argue that previous methods have focused on the energy difference between ID and OOD data, neglecting the energy differences among ID data samples. They introduce an energy regularization framework that unifies various long-tail recognition methods and extends it to OOD generalization scenarios. The paper empirically and theoretically demonstrates the effectiveness of the proposed method, called Influence-Aware Energy Regularization (IAER), through experiments on long-tail datasets, subpopulation shift benchmarks, and OOD generalization benchmarks", "questions": "1. In the transition from Eq. 5 to Eq. 6, the term $\\hat{\\beta_x} \\cdot E_\\theta(x)$ appears to be missing on the left side of the equation. Could the authors please clarify the derivation process between these two equations？Additionally, what does  $\\mathcal{L}$ refer to in Eq.6?  Is $\\mathcal{L}$ the same as $\\mathcal{L}_{ce}$ in Eq.5?\n\n2. Is the influence function calculated at each training epoch? Table 6 shows the time used for the influence function calculation, which appears to be computationally intensive. Could the authors provide a breakdown of the computational costs, specifically comparing the time spent on influence function calculation to the overall training time? This would give a clearer picture of the method's efficiency.\n\n3. In Tab. 3, the selection of the validation set is crucial for model performance, especially in long-tailed scenarios where the test distribution can significantly differ from the training distribution. Could the authors provide a more detailed explanation of their validation set selection strategy, including the rationale behind their choices for different datasets? In Appendix A.1, why is a balanced validation set maintained for CIFAR, and how does the number of data points in the validation set affect the IAER? For ImageNet-LT, the validation set samples are excluded from the training set, whether it break the imbalance factor? Furthermore, excluding certain specific classes will downgrade the corresponding performance in Tab. 3, decreasing the generality of the method.\n\n4. In line 317, the terms \"positive influence\" and \"negative influence\" are used. Could the authors provide clear definitions of these terms in the context of their method? Additionally, could they explain how these influences relate to the energy values of data points, particularly for low-energy points?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper offers a novel perspective on energy-based models by focusing on the energy differences among ID data samples, which is often overlooked in favor of the energy difference between ID and OOD data.\n2. It provides a unified framework that connects long-tail recognition methods like data re-weighting and margin control, interpreting them as implicit energy regularization techniques.\n3. The paper supports its claims with extensive experiments on various benchmarks, datasets, and scenarios, demonstrating the effectiveness of the proposed IAER method.", "weaknesses": "1. Inconsistent symbol usage can lead to ambiguity. For instance, in line 150, the variable $y$ denotes the specific label associated with a data point $x$, whereas in Equation 2, $y$ appears to represent the set of all training classes.\n2. The definition of symbols should be clear and unambiguous. For example, in line 210, it states \"for a data sample $x$ suppose the ground truth is $\\hat{y}$\".  The term\"suppose\" typically implies an assumption or prediction, which might be misleading in this context. Additionally, it is important to clarify the distinction between $y$ and  $\\hat{y}$.\n3. The baselines used in the study appear to be outdated. For instance, the baseline in Table 2 dates back to 2019, and the baseline in Table 3 is from 2020. Given the rapid advancements in the field, using more recent benchmarks would provide a more accurate and relevant comparison, thereby enhancing the validity and reliability of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MLd1lQEXbx", "reviewer": "Reviewer_aGfa", "summary": "This paper proposes a new approach to improve model performance in detecting and generalizing to out-of-distribution (OOD) data. Unlike traditional methods, which primarily focus on the energy difference between IID and OOD data, this approach also considers energy differences within IID samples. The authors explain that the re-weighting and margin-based methods commonly used in long-tailed learning implicitly apply energy regularization. Then the authors introduce Influence-Aware Energy Regularization (IAER), which adjusts energy values based on influence functions derived from training data. Empirical experiments demonstrate the effectiveness of this method.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The idea of considering energy differences among in-distribution data samples is both novel and sensible.\n- The authors conduct a series of experiments, spanning long-tailed datasets, subpopulation shift benchmarks, and OOD generalization benchmarks, to evaluate the effectiveness of the proposed method.", "weaknesses": "- The connection between Sec.3 and Sec.4 appears somewhat disjointed. The motivation for the proposed method in Sec.4 seems detached from Sec.3, lacking continuity and a strong foundation.\n\n- The explanation of the re-weighting and margin control methods as implicitly affecting training energy feels somewhat forced. For instance, the paper primarily uses the first term in Eq. (6) to justify the impact of $\\hat{\\beta}_x$. However, the second term in Eq. (6), which also contains $\\hat{\\beta}_x$, influences the final loss. Thus, the effectiveness of energy regularization remains somewhat ambiguous.\n\n- The method's overall effectiveness is limited. For example, in Tab.3, when IAER is combined with cRT, the performance across all classes declines, and with LWS, the improvement brought by IAER across all classes is only about 0.2%.\n\n- There are some typos, such as in Tab.3, where \"iNatrualist\" should be \"iNaturalist\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)