Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper has provided an analysis method to evaluate the feature representation of energy-based models based on feature diversity. The authors extend the probably approximately correct (PAC) theory in the view of redundancy reduction on the performance of energy-based models.
strength_and_weaknesses: This paper has provided a way to identify the quantification of feature diversity by the proposed measurement method named “θ-diversity”. The authors extend this idea in PAC learning and show that reducing the redundancy of the feature set can improve the generalization of EBMs. Please refer to the following questions about my concerns.

Weakness:
1. For the definition 1, which is the major contribution of this paper, the definition also relies on a probability value τ. Therefore, the method should at least be called ​​θ- τ (like epsilon-delta) to avoid confusion.
2. How to define high probability in definition 1? Is it related to the distribution of x. Should this be considered in the defined boundary?
What will happen if there are only a few samples that contain similar contents, where the boundary will be close to zero in most feature functions? How do we treat the effect of the distribution of data (x) on the boundary value?
3. In formula 16, the regularization term is not just about the boundary but an integration over the entire dataset and feature set. Even the experiment could sufficiently show improvement in performance, I can’t see its relationship with the proposed diversity measurement function.
4. In the image generation experiment (3.2), the authors only apply the proposed method on a simple dataset MINIST. There are lots of works based on EBM for image/text generation on a large dataset. The author should at least show some evaluation results to prove the effectiveness of the proposed method.
5. In table 1, it looks like the selection of beta could affect the performance. The author should also conduct an experiment on the effectiveness of the proposed method in terms of beta.
6. In 3.3, what are boundary settings?

Writing:

It would be better for the authors to reorganize the paper structure in section 1&2. The introduction of energy models applied in classification, regression, or implicit regression tasks should be put in the second part as related materials while a brief introduction about PAC learning should be put in the 1st section since it has a close relationship with the contribution.

Missing important references:

The description of the advance of EBMs in the first paragraph of the paper is incomplete and lacks important pioneering works. The authors randomly cite EBM works without mentioning those important ones.  For example, the first paper that proposes to train a generative EBMs parameterized by a modern deep neural network and learned it by Langevin based MLE is in (Xie. ICML 2016) [1]. The first shallow EBM using Langevin for data generation can date back to 1998 in [2]. After the era of deep learning, the paradigm in [1] has been applied to videos [3], 3D voxels [4], point clouds [5] and scenes [6]. Supervised approaches might include supervised conditional learning [7], saliency prediction [8], and trajectory prediction [9]. New generative EBM frameworks also include coarse-to-fine EBM [12], CoopNet [10], CoopFlow [11], VAEBM [13]. Without knowing the history and the advance of EBMs, the contribution of the paper might be questionable.

 
[1] A Theory of Generative ConvNet. ICML 2016. 

[2] Grade: Gibbs reaction and diffusion equations. ICCV 1998.

[3] Synthesizing Dynamic Pattern by Spatial-Temporal Generative ConvNet. CVPR 2017. 

[4] Learning Descriptor Networks for 3D Shape Synthesis and Analysis. CVPR 2018.

[5] Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification. CVPR 2021.

[6] Patchwise Generative ConvNet: Training Energy-Based Models from a Single Natural Image for Internal Learning. CVPR 2021.

[7] Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning. TPAMI 2021.

[8] Energy-Based Generative Cooperative Saliency Prediction. AAAI 2022

[9] Energy-Based Continuous Inverse Optimal Control. TNNLS 2022

[10] Cooperative Training of Descriptor and Generator Networks. PAMI 2018

[11] A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model. ICLR 2022.

[12] Learning Energy-Based Generative Models via Coarse-to-Fine Expanding and Sampling. ICLR 2021.

[13] VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models. ICLR 2021.
clarity,_quality,_novelty_and_reproducibility: Clarity and Quality: The materials in section 1&2 need to be re-organized.
Novelty: Moderately.
Reproducibility: Should be easy to reproduce.
summary_of_the_review: This paper has proposed a metric to define the diversity in feature representation and extend it in the loss function to improve the model generalization. However, the author hasn’t given enough proof and reason to support the effectiveness of the proposed method. Furthermore, as shown in formula (16), this could be used as a general term in supervision of deep learning models, but not necessarily to energy-based models. Besides, the experiments are only conducted on some simple dataset and are not sufficient to demonstrate the universe effectiveness of the proposed method.

--Post Rebuttal
I change my score from 3 -> 5 due to the explanation and revision from the author. Although the author has claimed that the experiment part is only to show how the proposed regularization term affects generalization in the EBM context, I still feel it necessary to see experiments on another task/dataset except for the CL (as proposed by reviewer mRZM). Moreover, if the author could also answer the concerns pointed out by reviewer AdvF and oTpp where the proposed regularization term will eventually reduce $\sqrt{\left(D A^2-v^2\right)}$, I'd like to further move my ratings.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper provides a new regulariser term for training energy-based models, which promotes feature diversity.
A theoretical analysis on the generalisation performance of energy-based models using the PAC-learning framework gives a solid motivating evidence on the need of the regularizer.
Specifically, the authors consider an existing generalisation bound for energy-models (where the difference between true and empirical averaged energy score is upper bounded by two terms, namely the Rademacher complexity of the energy function and its supremum value) and extend the theory by expressing the two terms in the bound as a function of the parameter involved in the new regulariser term.
The analysis is carried on three different kinds of energy functions for the purposes of regression, binary classification and implicit regression.

Experiments are performed:
- On a synthetic dataset showing the benefits of including the regulariser over unregularised approaches.
- On a MNIST task for implicit density estimation, thus evaluating generative and log-likelihood performance.
- On a continual learning task on CIFAR10 and CIFAR100, thus providing evidence on the improved predictive performance.

strength_and_weaknesses: **Strenghts**
- Regularization in energy-based models is an important and relevant problem.
- The proposed regulariser is to my knowledge new and can potentially link to recent work on redundancy reduction criteria used in self-supervised learning.
- The theory for the regulariser is simple and at the same time elegant and it provides a solid motivation on the need of the regulariser. Additionally, I think that the analysis can trigger new discussion in the community of energy-based models and inspire new regularising approaches.
- The paper is well-written and clear. I enjoyed reading it.

**Weaknesses**
- The claim that “the theory developed in our paper is agnostic to the loss function” is not correct (this appears in several parts of the paper). Indeed, note that the contributing term on the Rademacher complexity of the energy function in Theorem 2 (regression, using $L_1$ norm energy score) and in Theorem 3 (binary classification, using a cross-entropy-like energy score) doesn’t depend on the diversity parameter from the regulariser. Consequently, minimising the proposed regulariser doesn’t guarantee a reduction between true and empirical estimate of the average energy score for those two cases. On the contrary, it seems that the generalisation performance depend on the definition of the energy score function. Can you be more precise and elaborate on this aspect?
- The claim that “we provide theoretical guarantees for WORKS ON SELF-SUPERVISED LEARNING showing that reducing redundancy in the feature space can indeed improve the generalisation of the EBMs” is not precise. Indeed, note that the link between feature diversity and the objectives considered by negative-free approaches in self-supervised learning is not clear yet and it should be made more explicit. The latter approaches typically attempt to increase the correlation between representations of different views of the same data, while reducing their redundancy. It has been shown that this can be related to the principle of information bottleneck. Would it possible to make a clear link between the proposed diversity regulariser and the information bottleneck? This would strenghten the value of the proposed theory.
- Code is not available


**Minor questions**
- In proof of Lemma 4, shouldn’t you consider using the $L_2$ norm in order to be consistent with the definition of the energy function?
- The range of values for the hyper parameter beta in the regulariser is weird. Do you have an intuition on why you consider such small values? Should this range be considered also in other tasks)?

clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is clear and well-written. I enjoyed the reading.

**Quality**

The methodology used in the theoretical analysis is correct and the results are sound. However, some of the claims are not well supported.
The evaluation methodology is also technically correct.

**Novelty**

The paper proposes a new regulariser to promote feature diversity in energy-based models and an accompanying theory motivating its need.
The theory sheds new lights on the generalisation performance of energy-based models and it opens up to new directions for regularising such models. Additionally, it can potentially connect to recent work using the principle of redundancy minimisation in self-supervised learning.

**Reproducibility**

Code is not available.
summary_of_the_review: Overall, the paper provides a significant contribution to the literature of energy-based models. However, some of the claims are not well supported and should be made more precise. I initially recommend for a score of 6.

---- POST-REBUTTAL ----

While I think that the theory represents an interesting contribution of the work, the main concern on the bounds still remains. Indeed, the diversity parameter only controls one addend in the bounds of Eq. (13) and Eq. (14). Its maximization doesn't necessarily reduce the difference between expected and empirical energy scores. Additionally, as pointed out by reviewer AdvF, increasing the diversity paramater might indirectly increase the magnitude of the feature set, consequently increasing $A$ in the bounds and "cancelling its positive effect".
In my opinion, strengthening these aspects would suffice to accept the paper (even without an extensive experimental analysis). However, since the theory needs additional explanations/modifications, I consider the paper incomplete and not ready for publication yet. Therefore I change my score to 5.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper extends the PAC theory of EBMs to analyze the impact of feature diversity on the performance of EBMs. The generalization bounds for regression, binary classification and implicit regression w.r.t. the feature set redundancy are derived. Experimental results on MNIST for image generation and Cifar10/100 for continual learning are provided. 
strength_and_weaknesses: Strength:

1. The paper is well organized and well written.
2. The extension of the PAC theory of EBMs of Zhang et al. to feature diversity analysis is interesting and looks solid. But frankly, it’s hard for me to fully verify all the mathematical derivations given such a short ICLR review timeline. 

Weakness:

1. I have been working in this area for a while. It happened to me that I read a similar publication a year ago that has the same title (https://openreview.net/forum?id=ks3Q08yy66rv). The theory part of the current submission is almost the exact copy & paste of the workshop paper above. The only new addition is the experiment section. So, to my understanding, the contribution of this paper is mainly empirical study.

2. The experimental evaluations are very limited to small benchmarks and non-competitive baselines. For example, the image generation is only limited to MNIST without considering at least Cifar10/100/CelebA etc. In terms of the baseline algorithms, only the EBM algorithm of Du et al 2019 is compared and all latest SOTA EBM methods are ignored.

3. Although the authors show that the gains are consistent. But they seem very small. Even on the weak baseline of CL, where EBM-CL’s accuracies on CIFAR10/100 are about 30%-40%, the improvements are insignificant.

clarity,_quality,_novelty_and_reproducibility: The theory part of the current submission is almost the same as the workshop paper. The only new addition is the experiment section. So, to my understanding, the contribution is mainly empirical study, and the novelty is low.

The main algorithmic change is the regularization (Eq. 16), which is straightforward to implement. 

summary_of_the_review: I’d recommend a strong rejection of the paper. The reasons are two-fold:
1. The theory part is almost exact the same to the workshop paper of 2021.
2. The empirical evaluations are significantly weak.

Even the workshop paper can be claimed as an un-published work, the authors should at least cite it considering even Hinton’s RMSProp slides have been cited over 800 times.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a unifying PAC theory proof, showing the potential of improving the generalization error of energy-based approaches by reducing the redundancy of the feature set. It combines the Rademacher complexity and the definition of v-diversity to compute the upper-bound of the gap between the true and empirical expectation of the energy. Some experiments are conducted to verify its claims.
strength_and_weaknesses: Strength

- Well-organized paper with strong motivation.
- The proof framework is unifying, explicit, and easy to follow.
- The proposed simple regularization method is easy to be implemented and validated, and the experimental results are promising.

Weaknesses
- The empirical evidence is not strong enough to show a direct correlation between the generalization error of energy-based approaches and the v-diversity defined in the paper. On one hand, the proposed regularization algorithm is only tested on some simple image datasets, e.g. MNIST, and CIFAR. On the other hand, this paper only reports limited experimental results on image generation and continual learning, which are inconsistent with the theory part, including regression, classification, and implicit regression. Therefore, in order to prove the consistency of theory and practice, authors should either provide some PAC theory proof of image generation and continual learning or conduct experiments on the regression, classification, and implicit regression tasks and report the detailed results compared with standard energy-based approaches.
- The definition of v-diversity is not well explained. I do not think it can represent the diversity of the feature set since $\phi_i(x)$ and $\phi_j (x)$ are at different positions of the feature set.  Besides, in the paper, all the upper-bounds are correlated with $\sqrt{(DA^2-v^2 )}$ , and the author claims that we can deduce the upper-bounds by increasing $v$. However, as $v$ increases, $A$ also has an upward tendency since increasing $v$ could cause the increase in the magnitude of $\phi_i(x)$ . Therefore, I don 't think these upper-bounds can explain the role of v-diversity.

clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow, while the experiemental evaluations are weak and insufficient.
summary_of_the_review: The theory proposed in this paper lacks many important explanations, including definition and conclusion parts. Furthermore, the experimental tasks are inconsistent with the theory.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the influence of feature diversity on the generalization of energy-based models (EBM), which refers to the gap between the estimated energy function and the true energy distribution.  The authors propose to improve the performance of Energy-Based Models (EBMs) by enhancing the feature diversity of the inner model. It provides a PAC theory upper-bound (correlated with the feature diversity) of the gap between the true and empirical expectation of the energy for three kinds of tasks. To verify the effectiveness of the upper-bounds, it designs a simple regularization algorithm, which shows performance gain compared with standard EBMs.  
strength_and_weaknesses: ### Strength
- This paper studies the generalization ability of EBMs through the lens of feature diversity, which is important due to the wild applications of EBMs
- The proposed theory and method are clearly motivated and described.
- The theoretical proof is based on Rademacher complexity and is easy to follow.
- It successfully deduces the upper-bounds for three different forms of EBMs under the unifying framework and the experimental results are promising.  
### Weaknesses
- The deduced upper-bounds are typically loose and might not be able to guide the practice. Hence, it is better to have some theoretical or intuitive explanations about how tight those upper-bounds are.
- The experiments in this paper is kind of insufficient. This paper only includes some experimental results on image generation and continual learning, which are not consistent with its theoretical proof of regression, classification, and implicit regression tasks. Therefore, experimental results on the above three tasks should be reported clearly to verify the theory. Besides, the authors should conduct some experiments on more realistic and challenging datasets, such as ImageNet, CelebA etc.

- Comparison between prior work is lacked . For example, EBGAN [Zhao et al. 2017] uses a similar regularization based on cosine distance for diversity generation. 

- Validity of the diversity definition 1.   I am wondering whether Def. 1 is a good measure of diversity, especially when the diversity should be used for a set of samples $S=\{ x_1, … , x_n \}$, as indicated in Eq. 16.  In Eq. 16, you simply sum of the contributions of all samples. However, some holistic information might be lost in this way.  

There are well-established diversity tools in the literature, for example, the DPPs, see a nice survey and the references therein [Kulesza- Taskar 2012]. To apply DPP here,  for one feature $i$,  it is sensible to take the response of all n samples as its representation vector  $[\phi_i(x_1), … \phi_i(x_n)]^T$, then you can construct the DPP diversity metric by choose some kernel measuring the similarity. This might give rise a more reasonable diversity metric since DPP has a clear interpretation as the square of  volume spanned by the representation vectors. 

Can you comment on the above possibilities?   

- There seems to be some misuse of symbols in the paper. For example, it uses the sample symbol R_m to represent both empirical Rademacher complexity (in Definition 2) and the standard Rademacher complexity (in Lemma 2). The writting needs to be further checked and polished. 

### References

Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. ICLR, 2017.

Alex Kulesza, Ben Taskar, et al. Determinantal point processes for machine learning. Foundations and Trends® in Machine Learning, 5(2–3):123–286, 2012


clarity,_quality,_novelty_and_reproducibility: See the comments above
summary_of_the_review: This paper proposes an interesting theory analysis and improvement for EBMs by strengthening the feature diversity. However, considering the upper-bounds are loose and there are insufficient experimental results, the paper needs to be further improved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the influence of feature diversity on the energy-based models (EBM) and proposes to improve the performance of Energy-Based Models (EBMs) by enhancing its feature diversity. Specifically, the paper provides a PAC theory upper-bound of the gap between the true and empirical expectation of the energy for three kinds of tasks. To verify the effectiveness of the upper-bounds, the authors design a simple regularization algorithm to reduce the feature redundancy, and show performance gain compared with standard EBMs. The paper receives a total of 5 reviews.  The rebuttal hasn’t addressed the major concerns, including insufficient experiment (raised by Reviewers mRZM, VPp, and FAvoj) and math issue in the upper-bound (raised by Reviewers AdvF, oTpp, FAvo),  of the reviewers and all reviewers lean to reject the paper. The AC thinks that even though the paper studies an interesting and important problem for energy-based learning and the preliminary experiments also show some performance gain from the proposed method, there remains some major concerns that lead to a rejection to the current version.  Given the promising results of the current paper, the AC urges the authors to further improve their paper by taking into account all the suggestions provided by the reviewers, and then resubmit it to the next venue.

justification_for_why_not_higher_score: The paper cannot be accepted because of insufficient experiments and some math issues on the upper bounds.
Decision: Reject