On Feature Diversity in Energy-based Models
Authors: Firas Laakom
Jenni Raitoharju
Alexandros Iosifidis
Moncef Gabbouj
Venue: Submitted to ICLR 2023 (2023)
Keywords: energy-based models
redundancy reduction
feature diversity

Abstract: Energy-based learning is a powerful learning paradigm that encapsulates various discriminative and generative approaches. An energy-based model (EBM) is typically formed of inner-model(s) that learn a combination of the different features to generate an energy mapping for each input configuration. In this paper, we focus on the diversity of the produced feature set. We extend the probably approximately correct (PAC) theory of EBMs and analyze the effect of redundancy reduction on the performance of EBMs. We derive generalization bounds for various learning contexts, i.e., regression, classification, and implicit regression, with different energy functions and we show that indeed reducing redundancy of the feature set can consistently decrease the gap between the true and empirical expectation of the energy and boosts the performance of the model. 

Under review as a conference paper at ICLR 2023
ON FEATURE DIVERSITY IN ENERGY-BASED MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Energy-based learning is a powerful learning paradigm that encapsulates various discriminative and generative approaches. An energy-basee based model (EBM) is typi- cally formed of inner-model(s) that learn combination of the different features generate an energy mapping for each input configuration. In this paper, we focus the diversity of the produced feature set. We extend the probably approximately correct (PAC) theory of EBMs and analyze the effect of redundancy reduction on the performance of EBMs. We derive generalization bounds for various learning contexts, i.e., regression, classification, and implicit regression, with different en- ergy functions and we show that indeed reducing redundancy of the feature set can consistently decrease the gap between the true and empirical expectation of the energy and boosts the performance of the model.
INTRODUCTION
The energy-based learning paradigm was first proposed by Zhu & Mumford (1998): LeCun et al. (2006) as alternative probabilistic graphical models (Koller & Friedman 2009). As their name suggests, energy-based models (EBMs) map each input 'configuration" to a single scalar, called the "energy'. In the learning phase, the parameters of the model are optimized by associating the desired configurations with small energy values and the undesired ones with higher energy values (Kumar al., 22199 Song & Ermon 2019, Yang et al. 2016). In the inference phase, given an incomplete input configuration, the energy surface is explored to find the remaining variables which yield the lowest energy. EBMs encapsulate solutions to several supervised approaches (LeCun et al.. 2006
Fang & Liu, 2016) and unsupervised learning problems (Deng al. 20200] [Bakhtin et al. 2021
Zhao et al. 2020 Xu et al. 2022) and provide a common theoretical framework for many learning
models, including traditional discriminative (Zhai et al. 2016 Li et al.
2020 and generative (Zhu
& Mumfford 1998, Xie et al. 2017b Zhao et al. 2017 approaches.
Che et al.
2020 Khalifa et al. 2021
Formally, let us denote the energy function by E'h, æ, y). where Gw (x) represents the model with parameters W be optimized during training and æ, y are sets of variables. Figure I illustrates how classification, regression, and implicit regression can be expressed as EBMs. In Figure (a), a regression scenario is presented. The input æ, e.g., an image, is transformed using inner model (w) and its distance, to the second input y is computed yielding the energy function. A valid energy function this case can be the L1 or the L2 distance. In the binary classification case (Figure (b)), the energy can be defined as E(h, x, y) = gyG((z) In the implicit regression case (Figure [ (c)), we have two inner models and the energy can be defined as the L-2 distance between their outputs E(h, æ. y) = [[ G. (y) In the inference phase, given an input x, the label y can be obtained by solving the following optimization problem:
y" = arg min E(h, x. x.y).
(I)
y
An EBM typically relies on an inner model, i.e., Gw to generate the desired energy landscape [LeCun et al. 2006). Depending on the problem at hand, this function can be constructed as linear projection, a kernel method, or neural network and its parameters are optimized in a data-driven manner in the training phase. Formally, Gw (æ:) can be written as
D
Gw (z) = Σ wr)r(x),
(2)

---

Under review as a conference paper at ICLR 2023
E(h.x.y)
E(h,x.y)
E(h.x.y)
()G)(())))
-yG((X)
VallGw (xx
Gy!l1
Gw(x)
Gw(x)
Gy
G,A(y)
y
y
(a)
(b)
(c)
Figure 1: An illustration of energy-based models used to solve (a) regression problem (b) a binary classification problem (c) implicit regression problem.
)D(-)}} } is the feature set, which can be hand-crafted, separately trained from unlabeled data (Zhang & LeCun 2017), or modeled by a neural network and optimized in the training phase of the EBM model (Xie et al. 2016 Yu et al., 2020; Xie et al. 2021). In the rest of the paper, we assume that the inner models Gw defined in the energy-based learning system (Figure D are obtained as a weighted sum of different features as expressed in equation
where { (di())....,o)))
In (Zhang, 2013), it was shown that simply minimizing the empirical energy over the training data does not theoretically guarantee the minimization of the expected value of the true energy. Thus, developing and motivating novel regularization techniques is required (Zhang & LeCun 2017)
We argue that the quality of the feature set (o)()).... (D(})} plays a critical role in the overall performance of the global model. In this work, we extend the theoretical analysis of (Zhang 2013) and focus on the 'diversity" of this set and its effect on the generalization ability of the EBM models. Intuitively, it is clear that a less correlated set of intermediate representations is richer and thus able to capture more complex patterns in the input. Thus, it is important to avoid redundant features for achieving better performance. However, a theoretical analysis is missing. We start by quantifying r-diversity:
the diversity of a set of feature functions. To this end, we introduce
Definition 1 ((0- -)-iierrsity. A set of feature functions, {01(),, ,p(.)))()))))) } is called O-diverse, if there exists a constant E R, such that for every input æ we have
Σ -0()))))()))) 0² 1
with high probability T.
Intuitively, two feature maps (b(-) and (.)+) are non-redundant, they have different outputs for the same input with high probability. However, if, for example, the features extracted using neural network with a ReLU activation function, there a high probability that some of the features associated with the input will be zero. Thus, defining lower bound for the pair-wise diversity directly is impractical. Therefore, we quantify diversity as the lower-bound over the sum of the pair-wise distances of the feature maps as expressed in equation and i measures the diversity of Jndd
set.
In machine learning context, diversity has been explored in ensemble learning (Li al. 2012 Yu
let al. 2011
Li et al. 2017), sampling (Derezinski et al., 2019, Biyik et al. 2019), ranking (Wu al., 2019 Qin & Zhu (013), pruning (Singh et al. 2020 Lee et al.) 2020), and neural networks (Xie et al. 2015 Shen et al. 202T). In Xie et al. 2015 2017a). it was shown theoretically and experimentally that avoiding redundancy over the weights of a neural network using the mutual angles as diversity measure improves the generalization ability of the model. In this work, we explore a new line of research, where diversity is defined over the feature maps directly, using the ( -)diverrrsittt. in the context of energy-based learning. In (Zhao et al. 2017). similar idea was empirically explored. A "repelling regularizer" was proposed to force non redunddndd or orthogonal feature representations. Moreover, the idea of learning while avoiding redundancy has been used recently in the context of semi-supervised learning (Zbontar et al., 2022 Bardes et al. 202T). Re- ducing redundancy by minimizing the cross-correlation of features learned using a Siamese network
2

---

Under review as a conference paper at ICLR 2023
(Zbontar et al. 2021) empirically shown improve the generalization ability, yet theoretical analysis to prove this has so far been lacking.
In this paper, close the gap between empirical experience and theory. We theoretically study the generalization ability of EBMs in different learning contexts, i.c., regression, classification, implicit regression, and derive new generalization bounds using the (dv-st-divrrity providing theoretical guarantees that avoiding redundancy indeed improves the generalization ability of the model. The contributions of this paper can be summarized as follows:
We explore new line of research, where diversity is defined over the features representing the imput data and not over the model's parameters. To this end, we introduce (0 7)- diversity quantification of the diversity of given feature set.
We extend the theoretical analysis (Zhang 2013) and study the effect of avoiding redun- daney of a feature set on the generalization of HBMs (Lemmas 3 to 7 and Theorem 1 to 5).
We derive bounds for the expectation of the true energy in different learning contexts, i.e., regression, classification, and implicit regression, using different energy functions. Our analysis consistently shows that avoiding redundancy by increasing the diversity of the feature set can boost the performance of an EBM.
2 PAC-LEARNING OF EBMS WITH (0 DIIVERRRIY
In this section, derive qualitative justification for (0 (0--00--iii) -ivversiity using probably approximately
correct (PAC) learning (Valiant] 1984 Mohri al.
2018 et al. 2019). The PAC-based theory
for standard EBMs has been established in (Zhang 2013) First, start by defining Rademacher complexity:
Definition 2. (Bartlett & Mendelson 2002
Mohri el al.) 2018) For a given dataset with m samples S = {x,, 34 Th from distribution D and model space F X - R with single dimensional output, the Empirical Rademacher complexity Rm(F) (F) of the F defined as follows:
Rm,
Eg
sup Σ o ( SEF
(4)
1=1
where the Rademacher variables = for,... ame } are independent uniform random variables in {-1,1}.
The Rademacher complexity R. (F) is defined as the expectation of the Empirical Rademacher complexity over training set, i.e., (F) ES Rm (F)). Based on this quantity. (Bartleti Mendelson, 2002), several learning guarantees for EBMs have been shown (Zhang (013). We recall the following two lemmas related to the estimation error and the Rademacher complexity. In
Lemma 2 we present the principal PAC-learning bound for energy functions with finite outputs.
Lemma 1. WWlf 2018 For F E R. assume that g - F}. We have
R. Ly Lipshhitt continuous function
and A = [g
R., S Lg (F).
(5)
Lemma 2. (Zhang 2013) For a well-defined energy function E(h, x. y) over hypothesis class H. input set X and output set (LeCun et al. 2006), the following holds for all h in H. with a proba- bility of at least 1
E ((z,)>>[ EE x,
Σ (x.u)ES
E(h, x, y) + (E)
m
+ M log(2/8)
2m
where E is the energy function class defined as (E(,, 4)ih E H.) Rm (E) its Rademacher complexity, and M is the upper bound of E.
3

---

Under review as a conference paper at ICLR 2023
Lemma provides a generalization bound for EBMs with well-defined (non- -egaaivgt and bounded energy. The expected energy is bounded using the sum of three terms: The first term is the empirical expectation of energy over the training data, the second term depends the Rademacher complexity of the energy class, and the third term involves the number of the training data m and the upper- bound of the energy function M. This shows that merely minimizing the empirical expectation of energy, i.c., the first term, may not yield a good approximation of the true expectation. In (Zhang & LeCCun 2017). it has been shown that regularization using unlabeled data reduces the second and third terms leading to better generalization. In this work, we express these two terms using the (1) d)-iiersityy and show that employing diversity strategy may also decrease the gap between the true and empirical expectation of the energy. In Section 2.1 we consider the special case of regression and derive two bounds for two energy functions based on L, and L2 distances. In Section 2.2 we derive a bound for the binary classification task using as energy function E(h, 2z y) -yGw (#) (LoCun et al. 2006). In Section 2.3] we consider the case of implicit regression, which encapsulates difterent learning problems such as metric learning, generative models, and denoising (LeCun et al. 2006). For this case, we use the L2 distance between the inner models as the energy function. In the rest of the paper, we denote the generalization gap, (() (E((), 2; y)] by AD.s.. E. All the proofs presented in the supplementary material.
m L(a,y) E(h, x, y)
2.1 REGRESSION TASK
Regression can be formulated as an energy-based learning problem (Figure | (a)) using the inner model h(x) (w(x) Ee= (,):(x) w (xx. We assume that the feature set is posi- tive and well-defined over the input domain X, i.c., Vo E X' (())) S A. the hypothesis class can be defined as follows: H = {h(x) = Gw(x)= = E (x):(x) = w ((x) & e F. Væ |xx])] S A}. the output set y
{ob (+), ... ()(())))))))))))))).)))))))))))) } is i-diverse with a probability is bounded, i.c., y < B, and the feature set The two valid energy functions which can be used for regression are E2 x, 2 [l (z)) - y1l3 and F (h, æ, y) ||Gw vlla (LeCun et al. 2006). We study these two cases separately and show theoretically that for both energy functions avoiding redundancy improves generalization of the EBM model.
ENERGY FUNCTION: Eg
In this subsection, we present our theoretical analysis on the effect of diversity on the generalization ability of an EBM defined with the energy function Ez(h, æ, y) = 1/2 Gl 211.. We start by the following two Landas]Jand[4
Lemma 3. With probability of at least we have
sup [M(x)) I ||wl|s0 (DA2
W.W
Lemma 4. With probability of at least t, we have
sup
[1 x, b)1 < (1120|lo (DA2 - (2) + B)
(8)
m.y.h
Proof. We have sup (h(x) - y < sup..g........ (lh(x)) + (wl) = (1|w] 00 VDA= 22 B). Thus
supa.y. EE,, x, v)1 < 1 (1|w|| DA2 - 02 +3
Lemmas 3 and 4 bound the supremum of the output of the inner model and the energy function as a function of 9, respectively. As it been seen, both terms decreasing with respect diversity. Next, we bound the Rademacher complexity of the energy class, i.c., R.,. (E).
Lemma 5. With a probability of at least T, we have
(E) < 2D][w||l oz (w//////o//// bollooo VDA' 02) + B)Rm(F),
Lemma 5 expresses the bound of the Rademacher complexity of the energy class using the diversity constant and the Rademacher complexity of the features. Having expressed the different terms of Lemma 2 using diversity, we now present our main result for an energy- basel model trained defined using Eg. The main result is presented in Theorem
4

---

Under review as a conference paper at ICLR 2023
Theorem 1. For the energy function E(h, n, y) ( ullg over the input set X RN,,
hypothesis class H = {h(x) = Gw(x)= = Σ i=1 w; = u ((x) E Vz: ||0(z)]12 - A}. and output set y R. if the feature set (01(),... (.).)} is 0 d-viees with a probability т, with probability of at least (1 5)T, the following holds for all h in H:
S E S /D|/w//e (1/w)llo VDA 02 + B) Rym (F)
12/(2/0)
+ (1 v DA - 02 + B)² where B is the upper-bound y, i.e. y S B.
(10)
2m
Theorem [ express the special case of Lemma 2 using the (1 r)-diversity of the feature set {oi()), (.) As it can been seen, the bound of the generalization enror is inversely pro- portional to v². This theoretically shows that reducing redundancy, i.c.. increasing v. reduces the gap between the true and the empirical energies and improves the generalization performance of the EBMs.
ENERGY FUNCTION: E1
In this subsection, Eg (h, æz y) = |[G - yll1- Similar to the previous case, we start by deriving bounds for the energy function and the Rademacher complexity of the class using diversity in Lemmas and 7
consider the second case of regression using the energy function
Lemma 6. With a probability of at least T, we have
sup [E(h, x. y) S (1/21l>0 (/ DA2 -22+221 B).
(11)
æ.g.h
Lemma 7. With a probability of least T, have (E) S 2D| Rum (F).
(12)
Next, we derive the main result of the generalization of the EBMs defined using the energy function Ej. The main finding is presented in Theorem
Theorem 2. For the energy function E(h, x. y) 1|G (a) yll over the imput set X' RN
hypothesis class H = {h(x)= = Gw(x) = Σ1 w,o,(x) = w & E F, Vo [0(x)] s
A}. and output set y C. R. if the feature set (o1(-).... 2D(+)} is O d-diverse with a probability T, then with a probability of at least (1 - d)T, the following holds for all h in H:
log (2/0)
AD.SE < 4D|/w//lo Rm (F) + (1|2]l VollllV/i DA2 - 02 Bly
(13)
2m
where B is the upper-bound of Y, ie., y S B, Vy V.
Similar to Theorem I in Theorem we consistently find that the bound of the true expectation of the energy is a decreasing function with respect to O. This proves that for the regression task reducing redundancy can improve the generalization performance of the energy-based model.
2.2 BINARY CLASSIFIER
Here, consider the problem of binary classification, illustrated in Figure T (b). Using the i
same assumption as in regression for the inner model, i.c., h(=) Gw (z)
D (
w (xx), energy function of E(h, x, yy -yGww(z (LeCun al. | (006), and the (v diiversiyy
of the feature set, we express Lemma for this specific contiguration in Theorem 3
Theorem 3. For the energy function E(h, x, y) -yGw(x over the input set X E N. hypoth-
esis class H {((xz)
Gw(x) = Σ wxd((x) w (o=)(()) $ E Vz || < A},
and output set y C. R. if the feature set (01(.),... .)(())))...))))))) is d-diverse with probability then with probability of at least (1 8)T, the following holds for all H:
log log(2/8)
A < 4D||w||o R. (F) + || DA2 8
(14)
2m
Similar to the regression task, note that the upper-bound of the true expectation is a decreasing function with respect to the diversity term. Thus, less redundant feature set, i.c., higher 0, has a lower upper-bound for the true energy.
5

---

Under review as a conference paper at ICLR 2023
2.3 IMPLICIT REGRESSION
In this section, we consider the problem of implicit regression. This is a general formulation of different set of problems such as metric learning, where the goal is to learn distance function between two domains, image denoising, object detection as illustrated in (LeCun et al.. 2006), or semi-supervised learning (Zbontar et al.) 202I). This form of EBM (Figure I (c)) has two inner models, Gw(() (-) and (t), which can be equal or different according to the problem at hand. Here, we consider the general case, where the two models correspond to two different combinations D
of different features, i.e., G. (a)) = Σm D w; @i (x)))))))))))))(()) and GW (y) = w & (y). Thus, we have different (1- - 7)-diversity term for each set. The final result is presented in Theorem f
(1)
Theorem 4. For the energy function E(h, x. y) 211GW (a) - -GW (y)l over the input set X
RN,, hypothesis class H = (x) (x) - (x) = Eis w, 0. (000 (x) = w(1)² (z))))))))
GW W (v) = -i=1 w. Vi (y) = w((2 (v) (1))) E Fi, (12) E F2, ve: : (0)()())) S AO), Vy: 11 (g)l S A 3, and output set Y c RN,, if the feature set 10 (). & ())
(2) (2)
is (()) diverss with a probability Ti and the feature set (0f²) ()) (),,...
3 Deas ()) is -lvveree with
a probability T2. then with a probability of at least (1 8) the following holds for all h in H:
D S 8( B(VJJ++ vTa) (T)))())) 1000 (Fi)) + (22) ||0 1100) (mm
+(11 + +Ja) /log(2/0)
(15)
2m
where JI = 1|2 (00² - 0002) and J22 = =lw0(2) (2) 1 (D
A2)² -2))2 -
The upper-bound of the energy model depends on the diversity variable of both feature sets. More- over, we note that the bound for the implicit regression decreases proportionally to 8², as opposed to the classification case for example, where the bound is proportional to 0. Thus, we can conclude that reducing redundancy improves the generalization of EBM in the implicit regression context.
2.4 GENERAL DISCUSSION
We note that the theory developed in our paper (Theorems [ to is agnostic to the loss function (LeCun et al. 2006) the optimization strategy used (Kumar et al. 2019 Song & Ermon| 2019 Yu eu et al. 2020 Xu et al. 2022). We show that reducing the redundancy of the features consistently decreases the upper- bound of the true expectation of the energy and, thus, can boost the generaliza- tion performance of the energy-based based model. It also should be noted that A, i.c., the upper bound of the features and & are connected. But our findings can be interpreted follows: given two mod- els with the same value of A (maximum I.gnorm of the features), the model with higher diversity & has lower generalization bound and is likely to generalize better. We note that our analysis is independent of how the features are obtained, e.g., handcrafted or optimized. In fact, in the recent -fte--of-the-ar EBMs (Khalifa et al., [20TT] Bakhtin al.] [20TT] Yu et al., 2020), the features are typically parameterized using deep learning model and optimized during training. Our contribu- tion is twofold. First, we provide theoretical guarantees that reducing redundancy in the feature space can indeed improve the generalization of the FBM. This can pave the way toward providing theoretical guarantees for WORKS ON SUBEFSSSERRIIEE LEARNING using redundancy reduc- MonnZbotarr et al. (202 y Bardes et al. (021); Zhao et al. (2017). Second, our theory can be used to motivate novel redundancy reduction strategies, for example, in the form of regularization, to avoid learning redundant features. Such strategies can improve the performance of the model and improve generalization.
3 SIMPLE REGULARIZATION ALGGORITH
In general, theoretical generalization bounds can be too loose to be direct practical implications (Zhang et al., 2017; Neyshabur et al. 2017). However, they typically suggest a regularizer to pro-
mote some desired aspects of the hypothesis class (Xie et al. 2015 Li et al.) [019) Kawaguchi et al.)
2017). Accordingly, inspired by the theoretical analysis in Section
2
we propose a straightforward

---

Under review as a conference paper at ICLR 2023
112
COM11N
21001
0
01
6,0000500
C200720
C2000364
0000032
0016
Figure 2:
From left to right: (a): 2-D swiss roll ground truth distribution, (b) Distribution learned using standard EBM model, (b) Distribution learned with augmented loss using our regularizer. Relative to the ground truth, the are 0.27426 and 0.2733, respectively.
distance of the standard EBM distribution and ours
strategy to avoid learning redundant features by regularizing the model during the training using term inversely proportional to 9 r-diversity of the features. Given an EBM model with a learnable feature set { (-)(-)) and training set we propose to augment the original training loss L as follows:
Laug
ΣΣ #$1
(16)
where is a syper-parameter controlling the contribution of the second term the total loss. The additional term penalizes the similarities between the distinct features ensuring learning diverse and non-redundant mapping of the data. As result, this can improve the general performance of model.
3.1 ToY EXAMPLE
We test our regularization strategy first using a toy data. We use an EBM model to learn the distri- bution of 2-D Swiss roll illustrated in Figure (a). For the EBM, ral network composed of two intermediate layers with 1000 units and ReLu activations. We train the models using Stochastic Gradient Langevin Dynamies (SGLD) sampling and the contrastive divergence-like algorithm proposed in (Du & Mordatch, 2019). The total objective of the standard EBM is expressed as follows:
fully connected neu-
L
N Σ a(E(xt))+ E(a,))) + E(xt)) - E(x,))
(17)
where x denote positive samples and x, negative samples. We augment this loss using equa- tion 16 i.e., the features are the latent representations obtained at the last intermediate layer.
The distribution learned using both standard and the proposed approach are illustrated using the kernel density estimation (Trrell & Scott 1992) in Figure 2 As it can be boosts the performance of the EBM model. Indeed, by comparing the two learned distributions, the EBM trained with approach led better approximation of the ground-truth distribution and was able to better capture the tail of the distribution as opposed to the original EBM.
avoiding redundancy
3.2 IMAGE GENERATION EXAMPLE
Recently, there has been a high interest in using EBMs solve image/text generation tasks Du & Mordatch| (2019); Du et al. (002T); Khalifa et al. (202T): Deng et al. (2020). In this subsection, validate the proposed regularizer on the simple example of MNIST digits image generation, as in (Du & Mordatch, 2019). For the EBM model, we use simple CNN model composed of four convolutional layers followed by a linear layer. The training protocol is the same as in (UvA) Du & Mordatch] 2019), i.e., using Langevin dynamics Markov chain Monte Carlo (MCMC) and sampling buffer to accelerate training. The full details are available in the supplementary material. In this example, the features, i.e.. the latent representation obtained the last intermediate layer, are learned in an end-to-end way. We evaluate the performance our approach by augmenting the contrastive divergence loss using equation 16 penalize the feature redundancy.

---

Under review as a conference paper at ICLR 2023
9
32
64
96
128
160
192
224
256
32
96
128
160
192
224
256
6T
a
a
2
2
2
2
32
64
96
128
160
192
224
256
Generation iteration
Figure 3: Qualitative results of our approach (3 = 1e 10): : Few intermediate samples of the MCMC
sampling (Langevin Dynamics).
Approach
FID
NLL loss
EBM
0.01085 = 0.00037
0.71124 + 0.01901
ours (3 = 11)
0.01071 + 0.00040
0.71089 + 0.01106
ours (3 = 1e 12-
0.01040 = 0.00034
0.71052 + 0.01118
ours (3 13) 0.00985 0.00058
0.71076 = 0.01105
Table 1: Table of FID scores and negative log-likelihood (NLL) loss of different approaches for generations MNIST images. Each experiment was performed three times with different random seeds, the results are reported the mean/SEM over these runs.
We quantitatively evaluate image quality of EBMs with 'Fréchet Inception Distance' (FID) score (Heusel et al. 2017) and the negative log-likelihood (NLL) loss in Tor different values B. We note that obtain consistently better FID and NLL scores by penalizing the similarity of the learned features. The best performance is achieved by 3 = le 1e-13 which yields more than 10%, in terms of FID, improvement compared to the original EBM model. To gain insights into the visual performance of our approach, we plot few intermediate samples of the MCMC sampling (Langevin Dynamics). The results obtained by the EBM with 3 1e 13 are presented Figure 3 Initiating from random noise, MCMC obtains reasonable figures after only 64 steps. The digits get clearer and more realistic over the iterations. More results are presented the supplementary material.
3.3 CONTINUAL LEARNING EXAMPLE
In this subsection, we validate the proposed regularizer the Continual Learning (CL) problem. CL tackles the problem of catastrophic forgetting in deep learning models (Parisi et al. 2019 Li
Hoiem 2017 Shibata et al. 2021). Its main goal is solve several tasks sequentially without Forgetting knowledge Iearned from the past. So, a continual learner is expected to learn new task, crucially, without forgetting previous tasks. Recently, an EBM-based CL approach was proposed in (Li et al.) (220)) and led to superior results compared to standard approaches. We use the same models and the same experimental protocol used in (Li et al. 2020). However, here we focus only on the class-incremental learning task using CIFAR 10 and CIFARIOR We evaluate the performance of our proposed regularizer using both the boundary- aware and boundary-agnosse settings. As defined in (Li et al. 2020), the boundary- aware refers the situation where the sequence of the tasks has explicit separation between them which is known to the model. The boundary agnostic case refers to the situation where the data distributions gradually changes without notion of task boundaries.
8

---

Under review as a conference paper at ICLR 2023
Boundary-aware
Boundary-agnostic
Method
CIFARIO
CIFAR100
CIFARIO
CIFAR100
FBM
39.15 + 0.86
29.02 + 0.24
48.40 + 0.80
34.78 + 0.26
ours (5 = le 1e-11)
39.61 0.81
29.15 + 0.27
49.63 + 0.90
34.86 = 0.30
ours (3 1e 12)
40.64 + 0.79
29.38 = 0.21
50.25 + 0.63
35.20 0.23
ours (3 = le 19)
40.15 + 0.87
29.28 +: 0.28
50.20 + 0.94
35.03 = 0.21
Table 2: Evaluation of class-incremental learning on both the boundary-aware and boundary- agnostic setting on CIFARIO and CIFARI00 datasets. Each experiment was performed times with different random seeds, the results are reported as the mean/SEM over these runs.
EBM
EBM
1e-11)
le-11)
ours (3
1e-12)
ours 1e-12)
oura (3 16-13)
oura (8 16-13)
Classifiattio accurscy (16)
Classiiication accuracy (16
Tasks
Tasks
Figure 4: Average test classification accuracy vs number of observed tasks CIFAR 10 using the boundary-aware (left) and boundary-agnostic (right) setting. The results are averaged over ten ran- dom seeds.
Similar to Section 3.2 we consider 'features' the representation obtained by the last intermediate layer. The proposed regularizer is applied on top of this representation. In Table 2 we report the performance of the EBM trained using the original loss and using the loss augmented with our additional term for different values of f. As shown in Table penalizing feature similarity and promoting the diversity of the feature set boosts the performance the EBM model and consistently leads to superior accuracy for both datasets. In Figure 4 we display the accumulated classification accuracy, averaged over tasks, on the test set. Along the five tasks, our approach maintains higher classification accuracy than the standard EBM for both the boundary- aware and boundary-agnosti settings.
CONCLUSION
Energy-based learning is a powerful learning paradigm that encapsulates various discriminative and generative systems. An EBM is typically formed of one (or many) inner models which learn com- bination of different features to generate an energy mapping for each input configuration. In this paper. we introduced a feature diversity concept, i.c., (v 7)-diversity, and used it extend the PAC theory of EBMs. We derived different generalization bounds for various learning contexts, i.e., regression, classification, and implicit regression, with different energy functions and we con- sistently found that reducing the redundancy of the feature set can improve the generalization error of energy-based approaches. We also note that our theory is independent the loss function or the training strategy used to optimize the parameters of the EBM. This provides theoretical guarantees on learning via feature redundancy reduction. Our preliminary experimental results confirm that this is indeed promising research direction and can motivate developing other approaches to promoting the diversity of the feature set. Future direction include more extensive experimental evaluation of different feature redundancy reduction approaches.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Deep energy based generative models. http::///vallcenooebbb... het.e..............oo.eeeee.e..
readthedocs 1o/
fal/://sssaaattt2nneeeeooooooorr /tt.o/ooooo//tooooraaa///eeeeepp Modelssooellssse .hmml] Accessed: 2022-05-01.
Anton Bakhtin, Yuntian Deng, Sam Gross, Myle Ott, Aurello Ranzato, and Arthur Szlam. Residual energy-based models for text. JMLR, 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: aetiaacciiaaaaannoooovrineeeerne regularizatiinn for self-supervised learning. arXiv preprint arXiv:2105, 04906, 2021.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. JMLR, 2002.
Erdem Bıyık, Kenneth Wang, Nima Anari, and Dorsa Sadigh. Batch active learning using determi- nantal point processes. arXiv preprint arXiv:1906.0723 2019.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your gan is secretly an energy-based model and you should use discriminator driven latent sampling. arXiv preprint arXiv:2003.06060 2020.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy-based models for text generation. arXiv preprint arXiv:2004 /1714, 2020.
Michal Derezinski, Daniele Calandriello, and Michal Valko. Exact sampling of determinantal point processes with sublinear time preprocessing. In Advances in Neural Information Processing Sys- tems, 2019.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, 2019.
Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy-based models. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Yi Fang and Mengwen Liu. A unified energy-based framework for learning to rank. In ACM International Conference the Theory of Information Retrieval, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:
2017.
Muhammad Khalifa, Hady Elsahar, and Mare Dymetman. A distributional approach to controlled text generation. ICLR, 2021.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Rithesh Kumar, Sherjil Ozair. Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint arXiv:1901.08508 2019.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. tutorial on energy-based learning. Predicting structured data, 1, 2006.
Seunghyun Lee, Byeongho Heo, Jung-Woo Woo Ha, and Byung Cheol Song. Filter pruning and re- initialization via latent space clustering. IEEE Access, 8, 2020.
Jian Li, Yong Liu, Rong Yin. and Weiping Wang. Multi-class learning using unlabeled samples: Theory and algorithm. In ICAI, 2019.
10

---

Under review as a conference paper at ICLR 2023
Jianxin Li, Haoyi Zhou, Pengtao Xie, and Yingchun Zhang. Improving the generalization perfor- mance of multi-class svm via angular regularization. In IJCAI, 2017.
Nan Li, Yang Yu, and Zui-Hua Zhou. Diversity regularized ensemble pruning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2012.
Shuang Li, Yilun Du, Gido M van de Ven, Antonio Torralba, and Igor Mordatch. Energy-based
models for continual learning. arXiv preprint arXiv:2011.12216, 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE TPAMI, 2017.
Mehryar Mohri, Afshin Rostamizadch, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring gener- alization in deep learning. NIPS, 2017.
Genman I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Lijing Qin and Xiaoyan Zhu. Promoting diversity in recommendation by entropy regularizer. In IJCAI, 2013.
Dazhong Shen, Chuan Qin, Chao Wang, Hengshu Zhu, Enhong Chen, and Hui Xiong. Regularizing variational autoencoder with diversity and uncertainty awareness. In IJCAI, 2021.
Takashi Shibata, Go Irie, Daiki Ikami, and Yu Mitsuzumi. Learning with selective forgetting. In IJCAI, 2021.
Pravendra Singh, Vinay Kumar Verma, Piyush Rai, and Vinay Namboodiri. Leveraging filter corre- lations for deep model compression. In The IEEE Winter Conference on Applications of Computer Vision, 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Neurips, 2019.
George R Terrell and David W Scott. Variable kernel density estimation. The Annals of Statistics, 1992.
Leslie G Valiant. A theory of the learnable. Communications the ACM, 1984.
Michael M Wolf. Mathematical foundations of supervised learning, 2018.
Qiong Wu, Yong Liu, Chunyan Miao, Binqiang Zhao, Yin Zhao, and Lu Guan, Pd-gan: Adversarial learning for personalized diversity-promoting recommendation. In IJCAI, 2019. doi: 10.24963/ ijcai. 20199.30
Bo Xie, Yingyu Liang, and Le Song. Diverse neural network learns true target functions. In Artificial Intelligence and Statisties, 2017a.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference Machine Learning, pp. 2644, PMLR, 2016.
Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu. Synthesizing dynamic patterns by spatial- temporal generative convnet. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 7093-7101, 2017b.
Jianwen Xie, Yifei Xu, Zilong Zheng, Song- Lang-Chun Zhu, and Ying Nian Wu. Generative pointnet: Deep energy-based learning on unordered point sets for 3d generation. reconstruction and classifi cation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14976-14985, 2021.
Pengtao Xie, Yuntian Deng, and Eric Xing. On the generalization error bounds neural networks under diversity-inducing mutual angular regularization. arXiv preprint 1511v0 2015.

---

Under review as a conference paper at ICLR 2023
Yifei Xu, Jianwen Xie, Tianyang Zhao, Chris Baker, Yibiao Zhao, and Ying Nian Wu. Energy- based continuous inverse optimal control. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Xu Yang, Xin Geng, and Deyu Zhou. Sparsity conditional energy label distribution learning for age estimation. In IJCAI, 2016.
Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon. Training deep energy- bassed models with f-divergence minimization. In ICML, 2020.
Yang Yu, Yu-Feng Li, and Zhi-Hua Zhou. Diversity regularized machine. In International Joint Conference on Artificial Intelligence, 2011.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. /CML, 2021.
Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. In ICML. PMLR, 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Xiang Zhang. Pac-learning for energy-based models. PhD thesis, Citeseer, 2013.
Xiang Zhang and Yann LeCun. Universum prescription: Regularization using unlabeled data. In AAAI, 2017.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. ICLR, 2017.
Yang Zhao, Jianwen Xie, and Ping Li. Learning mnrgy-bbaed generative models via coarse-to-fine expanding and sampling. In International Conference on Learning Representations, 2020.
Song Chun Zhu and David Mumford. Grade: Gibbs reaction and diffusion equations. In Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271), pp. 847-854. IEEE, 1998.
APPENDIX
A PROOF OF LEMMA 3
Lemma With probability of at least r, we have
sup
[b(x)) < ||w|| -DA2 W
(18)
where A sup [|0(2)]|2-
Proof.
2
(²(()))
Σ (x)
5
Σ || m wwll (o
||w/|
1o0
2
Σ (x)
[=1
1|
Σ se (z)66)))))
= ||
Σ (20)
+
Σ (x)),,
(19)
We have
[2x < A. For the first term in equation 19 we have Σn, Im (z=) 22, By using
the identity (m) (x) 1 (xm(()) + (x)) rewritten as
(0)( 2n (2)))) the second term can be
Σ (m) = 1 Σ
Σ
(z)) + (x)) (dm )
(20)
mnkn
12

---

Under review as a conference paper at ICLR 2023
In addition, we have with probability 7, 2 2 (mm(((m(()) - =2(()))² 2 +. Thus, we have with probability at least
1 22(22220 1 (2( Σ Φm (z)) (a: -
- (D-1)44 - =
-202)
0².
(21)
mighn
By putting everything back to equation 19 we have with a probability
(x) S /
AA2 + (D 1)A4 0²) |w : (D.2 0
(22)
Thus, with a probability T,
sup [n(x)) S W
sup G (x) S |w| w >c V DA 92.
(23)
a.W
B
PROOF OF LEMMA 4
Lemma With probability of at least we have
1 sup [F(h, x. vil - / (1120//xc (DA2 02) + B)2.
(24)
m.y.h
Proof. We have supw.y.h [h(ze) - y/ < sup k. S.Px.y..(((((rr) + +wl) = (1/w|1>> VDA= - 00 B). Thus
supa,y,a (hh, x, b)1 S [(11w/] |o0 ADA 122 B)2.
C
PROOF OF LEMMA
Lemma With probability of at least we have
R.m (E) s
V (DA2 82) - B)Rm (F)
(25)
Proof. Using the decomposition property of the Rademacher complexity (if Φ is a L-Lipschitz function, then Rm (o(A)) I (A)) and given that 21l- ull is K-Lipschitz with stant K = suPa.u.h [1(ar)) yll & (1/ /DA² 82 + B). we have (E)(E) S (R),,,)) ([]wlloo / DA - 0 + B)R,(H)) where H (x) Σ i= w;)x(x) }. We also know that ||2/|| < D1|w0|lo0 Next, similar to the proof of Theorem 2.10 in (Wolf 2018), we note that >/1 (w) E <DI|wllas [F + +-FF)
G. where conv denotes the convex hull and is the set of d functions. Thus, Rm,(H) S (9) Dllwoll (conv(J (-F)) Dl|w]|oo Rm (F (-F)) = 2D/wwllo Rm (F).
D PROOF OF THEOREM
Theorem For the energy function E(h, x, y) = G - 1 v1ž. over the input set X E RN hypothesis class 7/ {h(x) Gw(x) ED (=))(x)) = ((x) E Vz [|0())]l S
A). and output set y C R, if the feature set (01(),... with probability of at least (1 8),, the following holds for all in H:
(n(-)} ..,,,(-))))))) is d-diverse with a probability т,
E ((th,))
1 m
Σ (m.y)s
E(h, x. v) 4 (1|wo)) v DA= 82 (F)
log(2/5)
2 (1|l v DA? - 02 + B)²
(26)
2m
where B is the upper-bound of V, i.c., y S B, Vy ).
Proof. We replace the variables in Lemma I using Lemma 4 and Lemma
13

---

Under review as a conference paper at ICLR 2023
E
PROOF OF LEMMA 6
Lemma With probability of least т, we have
sup (R, y)! s (1/0ll DDA DA2 02 + B).
(27)
a.y.h
Proof. We have y. [1(2)) - yl sup (. + (vl) (1/w0l00 D2
F PROOF OF LEMMA 7
Lemma With probability at least т, we have (E) s 2D|/w||ee Rm (F)
(28)
Proof. |. I-Lipschitz, Thus (E) & Rm (7)).
G PROOF OF THEOREM
Theorem For the energy function Ehh, x; y) || (x) - yll1. over the input set X' E RV hypothesis class H. (h(x) Gw Σb (2) w (xx) E F, Vo |1{())|l v
A). and output set y/ C R, if the feature set (0) (-) d-diverse with a probability T, then with a probability of at least (1 8)T, the following holds for all h in H:
E ((a.)>D(E(hh.)))
Σ (a.
F(h, x. v) 4D/ Rm
m
log(2/0) 2m
+ (1/w 0c V DA2 - 0 +B++BI/
(29)
where B is the upper-bound of y, i.e., y S B, y.
Proof. We replace the variables in Lemma 1 using Lemma 6 and Lemma 7.
H PROOF OF THEOREM 3
Lemma 8. With a probability of least T, we have
sup [E,,, x, v) s || DA? 12 w.g.h
(30)
Proof. We have sup yyG (x) S sup (w S | DA
L
Lemma 9, With probability of at least we have
Rm(EE) < 2D|[u/|ls Rm (F)
(31)
Proof. We note that for y E {-1,1}. a and -yo follow the same distribution. Thus, we have Rm (E) R., (H). Next, note that R,, (H) 2D w/lco Rm (F).
Theorem 3 For a well-defined energy function E(h, x, y) (LeCun et al. 2006). over hypothesis class H. input set X' and output set )2 if it has upper-bound M, then with probability of at least 1 - 6, the following holds for all h in H
(( 2, y)
Σ (s.y)ES
E(h, x. y) + +4D loo (F)
m
log(2/5) 2m
|//loo0 DA2 92
(32)
Proof. We replace the variables in Lemma 1 using Lemma 8 and Lemma 9.
14

---

Under review as a conference paper at ICLR 2023
1 PROOF OF THEOREM 4
Lemma 10. With a probability of at least T172, we have
sup |E(h, x. y) - (Jr + F2)
(33)
æ.g.h
Proof. We have [[G (x)
C (w)1 S 2(1GG( + ||G (1)) Similar to Theorem
(22)
1, we have sup || W (3:)11 I 11 (( A() - -m(1)) Ji and sup || (v)l
11 ( A ((3))) Jz. We also have E(h, y) [c) 0 (w)le O
Lemma 11. With a probability of at least 7172, we have
(m) S 4(VJI++ +
(100
Rom Rm) + 11c R1 Ru (F2))
(34)
Proof. Let f be the square function, i.e., f(x) 1 and En (o (x) GW (v) x e 3dd
X. E y)- We have & = f(Eo + (-Eo)). f Lipschitz over the input space, with a
constant I. bounded by sup,, w (W(x) + supy.w (WW()) (v) S VJ + VJ2. Thus, we have
Rm < (VJ + (mm (Eo + (-E6)) S 2(v + (m))))( Next, we note that
Rm (E)) (H, + (2H2)) Rm (H1) + Rm (H2). Using same as technique in Lemma
4, we have Rm (H1) S 2 (0 R Rm and Rm (H2) S 2D(2) ( Rm (F2).
O
Theorem 4 For the energy function E(h, v. = !
2 1 (z) G (1)ll,, over the input set
X E RN, hypothesis class H (GW (x) Ema wi oi (x) (()))((()) (W())
(1) (1)
(z)))))
D (2) (2)
wi of (g)
22)) T 212) (y) ((1) E Fi, (22)
F2, Vo |10 (z)]l2 S
1(1, Vy (1 S 1 and output set y c RN,, if the feature set (0f (),... Ba ()) is
g diverse with a probability Ti and the feature set {0( (.), & (-)) is div)-diverse with a
probability T2, then with probability of at least (1 - 5) T2. the following holds for all h in H
E ((a.)>D(E(0 x, y)1 S m
Σ
E(h, x. y)
(m.
+ +8( +8(VJ, + / Jz) (D (1 11 Rm (Fi) + 122) (2) Rm
Rm (F2))
+ (Ji + +T2) 1og(2/5)
(35)
2m
where Ji = 1| (DW AC1 - -000 000))) and Jz = 22) ( ( - _ -012)²)
Proof.
We replace the variables in Lemma 1 using Lemma 10 and Lemma 11.
J IMAGE GENERATION EXAMPLE SETTINGS AND ADDITIONAL RESULTS
For the EBM model, used simple CNN model composed of four convolutional layers followed by a linear layer. The full CNN model is presented in Table 3 The training protocol is the same as in (UvA) Du & Mordatch 2019) i.e., using Langevin dynamics MCMC and sampling buffer to accelerate training. All models were trained for 60 epochs using Adam optimizer with learning rate 1e 4 and a batch size of 128. In addition to the results presented in the paper, Figure 5 presents additional qualitative results. For the first two examples (top ones), the model is able to converge to a realistic image within reasonable amount of iterations. For the last two examples (in the bottom), we present failure cases of our approach. For these two tests, the generated image still improves over iterations. However, the model failed to converge to clear realistic MNIST image after 256 steps.
15

---

Under review as a conference paper at ICLR 2023
Layer
Output shape
Input
[1,28,28]
Cov (16 x 5)
[16,16,16]
Swish activation
[16,16,16]
Cov (32 x 3)
[32,8,8]
Swish activation
[32,8,8]
Cov (64 3 x 3)
[64,4,4]
Swish activation
[64,4,4]
Cov (64 x 3)
[64.2,2]
Swish activation
[64,2,2]
Flatten
[256]
Linear
[64]
Swish activation*
[64]
Linear
[1]
Table 3: Simple CNN model used in the example. * refers to the features' layer.
9
8
8
8
8
8
32
64
96
128
160
192
224
256
7
7
2
2
2
2
32
64
96
128
160
192
224
256
32
128
160
224
256
2
2
32
64
128
160
224
256
Generation iteration
Figure 5: Qualitative results of EBM augmented with our regularizer with 3 le 13, Few intermediate samples of the MCMC sampling (Langevin Dynamics).
16


Reviews:
[{"review_id": "rZ_cp6yA0T7", "reviewer": "Reviewer_FAvo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper has provided an analysis method to evaluate the feature representation of energy-based models based on feature diversity. The authors extend the probably approximately correct (PAC) theory in the view of redundancy reduction on the performance of energy-based models.", "summary_of_the_review": "This paper has proposed a metric to define the diversity in feature representation and extend it in the loss function to improve the model generalization. However, the author hasn’t given enough proof and reason to support the effectiveness of the proposed method. Furthermore, as shown in formula (16), this could be used as a general term in supervision of deep learning models, but not necessarily to energy-based models. Besides, the experiments are only conducted on some simple dataset and are not sufficient to demonstrate the universe effectiveness of the proposed method.\n\n--Post Rebuttal\nI change my score from 3 -> 5 due to the explanation and revision from the author. Although the author has claimed that the experiment part is only to show how the proposed regularization term affects generalization in the EBM context, I still feel it necessary to see experiments on another task/dataset except for the CL (as proposed by reviewer mRZM). Moreover, if the author could also answer the concerns pointed out by reviewer AdvF and oTpp where the proposed regularization term will eventually reduce $\\sqrt{\\left(D A^2-v^2\\right)}$, I'd like to further move my ratings.", "strength_and_weaknesses": "This paper has provided a way to identify the quantification of feature diversity by the proposed measurement method named “θ-diversity”. The authors extend this idea in PAC learning and show that reducing the redundancy of the feature set can improve the generalization of EBMs. Please refer to the following questions about my concerns.\n\nWeakness:\n1. For the definition 1, which is the major contribution of this paper, the definition also relies on a probability value τ. Therefore, the method should at least be called ​​θ- τ (like epsilon-delta) to avoid confusion.\n2. How to define high probability in definition 1? Is it related to the distribution of x. Should this be considered in the defined boundary?\nWhat will happen if there are only a few samples that contain similar contents, where the boundary will be close to zero in most feature functions? How do we treat the effect of the distribution of data (x) on the boundary value?\n3. In formula 16, the regularization term is not just about the boundary but an integration over the entire dataset and feature set. Even the experiment could sufficiently show improvement in performance, I can’t see its relationship with the proposed diversity measurement function.\n4. In the image generation experiment (3.2), the authors only apply the proposed method on a simple dataset MINIST. There are lots of works based on EBM for image/text generation on a large dataset. The author should at least show some evaluation results to prove the effectiveness of the proposed method.\n5. In table 1, it looks like the selection of beta could affect the performance. The author should also conduct an experiment on the effectiveness of the proposed method in terms of beta.\n6. In 3.3, what are boundary settings?\n\nWriting:\n\nIt would be better for the authors to reorganize the paper structure in section 1&2. The introduction of energy models applied in classification, regression, or implicit regression tasks should be put in the second part as related materials while a brief introduction about PAC learning should be put in the 1st section since it has a close relationship with the contribution.\n\nMissing important references:\n\nThe description of the advance of EBMs in the first paragraph of the paper is incomplete and lacks important pioneering works. The authors randomly cite EBM works without mentioning those important ones.  For example, the first paper that proposes to train a generative EBMs parameterized by a modern deep neural network and learned it by Langevin based MLE is in (Xie. ICML 2016) [1]. The first shallow EBM using Langevin for data generation can date back to 1998 in [2]. After the era of deep learning, the paradigm in [1] has been applied to videos [3], 3D voxels [4], point clouds [5] and scenes [6]. Supervised approaches might include supervised conditional learning [7], saliency prediction [8], and trajectory prediction [9]. New generative EBM frameworks also include coarse-to-fine EBM [12], CoopNet [10], CoopFlow [11], VAEBM [13]. Without knowing the history and the advance of EBMs, the contribution of the paper might be questionable.\n\n \n[1] A Theory of Generative ConvNet. ICML 2016. \n\n[2] Grade: Gibbs reaction and diffusion equations. ICCV 1998.\n\n[3] Synthesizing Dynamic Pattern by Spatial-Temporal Generative ConvNet. CVPR 2017. \n\n[4] Learning Descriptor Networks for 3D Shape Synthesis and Analysis. CVPR 2018.\n\n[5] Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification. CVPR 2021.\n\n[6] Patchwise Generative ConvNet: Training Energy-Based Models from a Single Natural Image for Internal Learning. CVPR 2021.\n\n[7] Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning. TPAMI 2021.\n\n[8] Energy-Based Generative Cooperative Saliency Prediction. AAAI 2022\n\n[9] Energy-Based Continuous Inverse Optimal Control. TNNLS 2022\n\n[10] Cooperative Training of Descriptor and Generator Networks. PAMI 2018\n\n[11] A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model. ICLR 2022.\n\n[12] Learning Energy-Based Generative Models via Coarse-to-Fine Expanding and Sampling. ICLR 2021.\n\n[13] VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models. ICLR 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hVzOEBtgthz", "reviewer": "Reviewer_AdvF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a unifying PAC theory proof, showing the potential of improving the generalization error of energy-based approaches by reducing the redundancy of the feature set. It combines the Rademacher complexity and the definition of v-diversity to compute the upper-bound of the gap between the true and empirical expectation of the energy. Some experiments are conducted to verify its claims.", "summary_of_the_review": "The theory proposed in this paper lacks many important explanations, including definition and conclusion parts. Furthermore, the experimental tasks are inconsistent with the theory. ", "strength_and_weaknesses": "Strength\n\n- Well-organized paper with strong motivation.\n- The proof framework is unifying, explicit, and easy to follow.\n- The proposed simple regularization method is easy to be implemented and validated, and the experimental results are promising.\n\nWeaknesses\n- The empirical evidence is not strong enough to show a direct correlation between the generalization error of energy-based approaches and the v-diversity defined in the paper. On one hand, the proposed regularization algorithm is only tested on some simple image datasets, e.g. MNIST, and CIFAR. On the other hand, this paper only reports limited experimental results on image generation and continual learning, which are inconsistent with the theory part, including regression, classification, and implicit regression. Therefore, in order to prove the consistency of theory and practice, authors should either provide some PAC theory proof of image generation and continual learning or conduct experiments on the regression, classification, and implicit regression tasks and report the detailed results compared with standard energy-based approaches.\n- The definition of v-diversity is not well explained. I do not think it can represent the diversity of the feature set since $\\phi_i(x)$ and $\\phi_j (x)$ are at different positions of the feature set.  Besides, in the paper, all the upper-bounds are correlated with $\\sqrt{(DA^2-v^2 )}$ , and the author claims that we can deduce the upper-bounds by increasing $v$. However, as $v$ increases, $A$ also has an upward tendency since increasing $v$ could cause the increase in the magnitude of $\\phi_i(x)$ . Therefore, I don 't think these upper-bounds can explain the role of v-diversity.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dGUnfAvm-v", "reviewer": "Reviewer_oTpp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides a new regulariser term for training energy-based models, which promotes feature diversity.\nA theoretical analysis on the generalisation performance of energy-based models using the PAC-learning framework gives a solid motivating evidence on the need of the regularizer.\nSpecifically, the authors consider an existing generalisation bound for energy-models (where the difference between true and empirical averaged energy score is upper bounded by two terms, namely the Rademacher complexity of the energy function and its supremum value) and extend the theory by expressing the two terms in the bound as a function of the parameter involved in the new regulariser term.\nThe analysis is carried on three different kinds of energy functions for the purposes of regression, binary classification and implicit regression.\n\nExperiments are performed:\n- On a synthetic dataset showing the benefits of including the regulariser over unregularised approaches.\n- On a MNIST task for implicit density estimation, thus evaluating generative and log-likelihood performance.\n- On a continual learning task on CIFAR10 and CIFAR100, thus providing evidence on the improved predictive performance.\n", "summary_of_the_review": "Overall, the paper provides a significant contribution to the literature of energy-based models. However, some of the claims are not well supported and should be made more precise. I initially recommend for a score of 6.\n\n---- POST-REBUTTAL ----\n\nWhile I think that the theory represents an interesting contribution of the work, the main concern on the bounds still remains. Indeed, the diversity parameter only controls one addend in the bounds of Eq. (13) and Eq. (14). Its maximization doesn't necessarily reduce the difference between expected and empirical energy scores. Additionally, as pointed out by reviewer AdvF, increasing the diversity paramater might indirectly increase the magnitude of the feature set, consequently increasing $A$ in the bounds and \"cancelling its positive effect\".\nIn my opinion, strengthening these aspects would suffice to accept the paper (even without an extensive experimental analysis). However, since the theory needs additional explanations/modifications, I consider the paper incomplete and not ready for publication yet. Therefore I change my score to 5.", "strength_and_weaknesses": "**Strenghts**\n- Regularization in energy-based models is an important and relevant problem.\n- The proposed regulariser is to my knowledge new and can potentially link to recent work on redundancy reduction criteria used in self-supervised learning.\n- The theory for the regulariser is simple and at the same time elegant and it provides a solid motivation on the need of the regulariser. Additionally, I think that the analysis can trigger new discussion in the community of energy-based models and inspire new regularising approaches.\n- The paper is well-written and clear. I enjoyed reading it.\n\n**Weaknesses**\n- The claim that “the theory developed in our paper is agnostic to the loss function” is not correct (this appears in several parts of the paper). Indeed, note that the contributing term on the Rademacher complexity of the energy function in Theorem 2 (regression, using $L_1$ norm energy score) and in Theorem 3 (binary classification, using a cross-entropy-like energy score) doesn’t depend on the diversity parameter from the regulariser. Consequently, minimising the proposed regulariser doesn’t guarantee a reduction between true and empirical estimate of the average energy score for those two cases. On the contrary, it seems that the generalisation performance depend on the definition of the energy score function. Can you be more precise and elaborate on this aspect?\n- The claim that “we provide theoretical guarantees for WORKS ON SELF-SUPERVISED LEARNING showing that reducing redundancy in the feature space can indeed improve the generalisation of the EBMs” is not precise. Indeed, note that the link between feature diversity and the objectives considered by negative-free approaches in self-supervised learning is not clear yet and it should be made more explicit. The latter approaches typically attempt to increase the correlation between representations of different views of the same data, while reducing their redundancy. It has been shown that this can be related to the principle of information bottleneck. Would it possible to make a clear link between the proposed diversity regulariser and the information bottleneck? This would strenghten the value of the proposed theory.\n- Code is not available\n\n\n**Minor questions**\n- In proof of Lemma 4, shouldn’t you consider using the $L_2$ norm in order to be consistent with the definition of the energy function?\n- The range of values for the hyper parameter beta in the regulariser is weird. Do you have an intuition on why you consider such small values? Should this range be considered also in other tasks)?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_ZH1KUrtW9", "reviewer": "Reviewer_VPpj", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper extends the PAC theory of EBMs to analyze the impact of feature diversity on the performance of EBMs. The generalization bounds for regression, binary classification and implicit regression w.r.t. the feature set redundancy are derived. Experimental results on MNIST for image generation and Cifar10/100 for continual learning are provided. ", "summary_of_the_review": "I’d recommend a strong rejection of the paper. The reasons are two-fold:\n1. The theory part is almost exact the same to the workshop paper of 2021.\n2. The empirical evaluations are significantly weak.\n\nEven the workshop paper can be claimed as an un-published work, the authors should at least cite it considering even Hinton’s RMSProp slides have been cited over 800 times.\n", "strength_and_weaknesses": "Strength:\n\n1. The paper is well organized and well written.\n2. The extension of the PAC theory of EBMs of Zhang et al. to feature diversity analysis is interesting and looks solid. But frankly, it’s hard for me to fully verify all the mathematical derivations given such a short ICLR review timeline. \n\nWeakness:\n\n1. I have been working in this area for a while. It happened to me that I read a similar publication a year ago that has the same title (https://openreview.net/forum?id=ks3Q08yy66rv). The theory part of the current submission is almost the exact copy & paste of the workshop paper above. The only new addition is the experiment section. So, to my understanding, the contribution of this paper is mainly empirical study.\n\n2. The experimental evaluations are very limited to small benchmarks and non-competitive baselines. For example, the image generation is only limited to MNIST without considering at least Cifar10/100/CelebA etc. In terms of the baseline algorithms, only the EBM algorithm of Du et al 2019 is compared and all latest SOTA EBM methods are ignored.\n\n3. Although the authors show that the gains are consistent. But they seem very small. Even on the weak baseline of CL, where EBM-CL’s accuracies on CIFAR10/100 are about 30%-40%, the improvements are insignificant.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "YT-dp8PCIN", "reviewer": "Reviewer_mRZM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the influence of feature diversity on the generalization of energy-based models (EBM), which refers to the gap between the estimated energy function and the true energy distribution.  The authors propose to improve the performance of Energy-Based Models (EBMs) by enhancing the feature diversity of the inner model. It provides a PAC theory upper-bound (correlated with the feature diversity) of the gap between the true and empirical expectation of the energy for three kinds of tasks. To verify the effectiveness of the upper-bounds, it designs a simple regularization algorithm, which shows performance gain compared with standard EBMs.  ", "summary_of_the_review": "This paper proposes an interesting theory analysis and improvement for EBMs by strengthening the feature diversity. However, considering the upper-bounds are loose and there are insufficient experimental results, the paper needs to be further improved. ", "strength_and_weaknesses": "### Strength\n- This paper studies the generalization ability of EBMs through the lens of feature diversity, which is important due to the wild applications of EBMs\n- The proposed theory and method are clearly motivated and described.\n- The theoretical proof is based on Rademacher complexity and is easy to follow.\n- It successfully deduces the upper-bounds for three different forms of EBMs under the unifying framework and the experimental results are promising.  \n### Weaknesses\n- The deduced upper-bounds are typically loose and might not be able to guide the practice. Hence, it is better to have some theoretical or intuitive explanations about how tight those upper-bounds are.\n- The experiments in this paper is kind of insufficient. This paper only includes some experimental results on image generation and continual learning, which are not consistent with its theoretical proof of regression, classification, and implicit regression tasks. Therefore, experimental results on the above three tasks should be reported clearly to verify the theory. Besides, the authors should conduct some experiments on more realistic and challenging datasets, such as ImageNet, CelebA etc.\n\n- Comparison between prior work is lacked . For example, EBGAN [Zhao et al. 2017] uses a similar regularization based on cosine distance for diversity generation. \n\n- Validity of the diversity definition 1.   I am wondering whether Def. 1 is a good measure of diversity, especially when the diversity should be used for a set of samples $S=\\{ x_1, … , x_n \\}$, as indicated in Eq. 16.  In Eq. 16, you simply sum of the contributions of all samples. However, some holistic information might be lost in this way.  \n\nThere are well-established diversity tools in the literature, for example, the DPPs, see a nice survey and the references therein [Kulesza- Taskar 2012]. To apply DPP here,  for one feature $i$,  it is sensible to take the response of all n samples as its representation vector  $[\\phi_i(x_1), … \\phi_i(x_n)]^T$, then you can construct the DPP diversity metric by choose some kernel measuring the similarity. This might give rise a more reasonable diversity metric since DPP has a clear interpretation as the square of  volume spanned by the representation vectors. \n\nCan you comment on the above possibilities?   \n\n- There seems to be some misuse of symbols in the paper. For example, it uses the sample symbol R_m to represent both empirical Rademacher complexity (in Definition 2) and the standard Rademacher complexity (in Lemma 2). The writting needs to be further checked and polished. \n\n### References\n\nJunbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. ICLR, 2017.\n\nAlex Kulesza, Ben Taskar, et al. Determinantal point processes for machine learning. Foundations and Trends® in Machine Learning, 5(2–3):123–286, 2012\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject