Title: Gradient-Guided Importance Sampling for Learning Discrete Energy-Based Models
Authors: Meng Liu
Haoran Liu
Shuiwang Ji
Venue: ICLR 2022
Keywords: Discrete energy-based models
ratio matching
importance sampling
gradient

Abstract: Learning energy-based models (EBMs) is known to be difficult especially on discrete data where gradient-based learning strategies cannot be applied directly. Although ratio matching is a sound method to learn discrete EBMs, it suffers from expensive computation and excessive memory requirement, thereby resulting in difficulties for learning EBMs on high-dimensional data. In this study, we propose ratio matching with gradient-guided importance sampling (RMwGGIS) to alleviate the above limitations. Particularly, we leverage the gradient of the energy function w.r.t. the discrete data space to approximately construct the provable optimal proposal distribution, which is subsequently used by importance sampling to efficiently estimate the original ratio matching objective. We perform experiments on density modeling over synthetic discrete data and graph generation to evaluate our proposed method. The experimental results demonstrate that our method can significantly alleviate the limitations of ratio matching and perform more effectively in practice.


Review discussion:
## Strength:
The paper is well motivated, clearly written, and seems technically sound. The paper does a good job in analyzing the limitations of ratio matching in expensive computation and excessive memory requirements, and in describing how it can leverage the gradient of the energy w.r.t. the discrete data space to address these limitations. The idea is simple, but can potentially be very useful in making ratio matching more practical for learning discrete energy-based models.
## Weaknesses:
- While the paper demonstrates improvements over vanilla ratio matching, it lacks comparison with other types of methods for learning discrete energy-based models.
For example, maximum-likelihood training with MCMC sampling (e.g. with Gibbs sampling or the more advanced Gibbs-with-gradients, which the authors acknowledge is a big inspiration for RMwGGIS), can be easily applied to the same problems, and should be included as additional baselines.
While the improvements over ratio matching are nice, if RMwGGIS cannot outperform other readily available methods, then it picked the wrong method to improve upon, and its improvements over ratio matching cannot be justified.
- I am not convinced by the current arguments in the section **Better optimization? ** in Sec. 3. 3, and I don't think this point is verified empirically as the authors claim. As the authors mention at the end of page 2, the estimator given by Eq. (3)/(4) is consistent, and if Eq. (3)/(4) is minimized perfectly, the obtained model distribution will capture the data distribution exactly. The authors claim that RMwGGIS, with a biased estimator, performs better than ratio matching due to better optimization. Does that mean that the theta learned with the biased objective function (12) actually results in lower value for the objective function in Eq.
(3)/(4)? I. e. somehow optimizing a different objective function (Eq. (12)) leads to better optimization of Eq. (3)/(4) than directly optimizing Eq. (3)/(4)? I would like to see this confirmed in experiments (which is a very simple evaluation). If it's really the case, I would like to see some analysis on why this is the case, since Eq. (12) is picked somewhat randomly, without theoretical justification as Eq. (7). If this is not the case, then there is no better optimization, which means Eq. (3)/(4) is not a good objective function, and the paper should rethink its foundation, or explain where the better performance comes from.
NO.
**Why can RWwGGIS be optimized better?** In Section 3.3, we provide two properties to intuitively understand this. (1) RMwGGIS introduces stochasticity; (2) RMwGGIS focuses on neighbors with low energies. **Here, we provide an additional insight to understand why the second property can lead to better optimization, by connecting it with hard sample mining.**
Hard sample mining [1,2] has been widely applied to train deep neural networks [3]. Our RMwGGIS is particularly highly related to hard negative training strategies. The basic idea for hard negative mining is to pay more attention to hard negative samples during training, which can usually achieve better performance since it can reduce false positives. In our setting of discrete EBMs, each training sample $x$ is a positive sample, and its energy should be pushed down. For each positive sample $x$, all $x_{-i}$ for $i=1,2,\cdots,d$ are negative samples, and their energy should be pushed up. Our RMwGGIS with the specific proposal distribution shown in Eq. (11) can **approximately** (will be clarified below) choose the $x_{-i}$’s that currently have low energies with larger probabilities. This has the same philosophy as hard negative mining.
Specifically, in our case, $x_{-i}$’s with low energies are hard negative samples since they are the most offending terms, which are close to the positive sample $x$ and have low energies.
Since our proposal distribution defined in Eq. (11) approximates the provable optimal proposal distribution given in Theorem 1, our proposal distribution thus approximately performs “hard negative mining”. The natural follow-up question is **how accurate is the approximation and how does it affect the learning process?** We answer this question to show that our RMwGGIS is technically sound. Please refer to **Appendix C** in our revised version for a detailed explanation with figure illustrations.
Thank you again for raising these interesting and insightful questions! We have included the above empirical results and extensive analysis in our revised version and the changes are highlighted in red color (Most changes are in Appendix C and E). Please let us know if there are any further questions. Please kindly consider improving the rating score if your concerns are addressed.
[1] Felzenszwalb, Pedro F., et al. "Object detection with discriminatively trained part-based models." IEEE transactions on pattern analysis and machine intelligence 32.9 (2009): 1627-1645.
[2] Rowley, Henry A., Shumeet Baluja, and Takeo Kanade. "Neural network-based face detection." IEEE Transactions on pattern analysis and machine intelligence 20.1 (1998): 23-38.
[3] Shrivastava, Abhinav, Abhinav Gupta, and Ross Girshick. "Training region-based object detectors with online hard example mining." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
Thank you for the constructive review. We are encouraged that you find that our work is well motivated and technically sound. We address the raised concerns as follows.
> While the paper demonstrates improvements over vanilla ratio matching, it lacks comparison with other types of methods for learning discrete energy-based models.
For example, maximum-likelihood training with MCMC sampling (e.g. with Gibbs sampling or the more advanced Gibbs-with-gradients, which the authors acknowledge is a big inspiration for RMwGGIS), can be easily applied to the same problems, and should be included as additional baselines.
While the improvements over ratio matching are nice, if RMwGGIS cannot outperform other readily available methods, then it picked the wrong method to improve upon, and its improvements over ratio matching cannot be justified.
Thank you for this suggestion. In our revised version. we add EBM (GWG) as suggested as a baseline in our graph generation experiment. The comparison is included in the following table. It can be observed that ratio matching and our RMwGGIS perform better than EBM (GWG), an MCMC-based maximum-likelihood training method, on this graph generation task.
| **Method**| *Degree* |*Cluster* |*Orbit* |*avg.* |
| ----------- | ----------- | ----------- |----------- |----------- |
| EBM (GWG)| 0.093|0.027|0.053|0.058|
| Ratio Matching| 0.062|0.066|**0.008**|0.045|
| **RMwGGIS**| **0.044**|**0.059**|0.013|**0.039**|
> I am not convinced by the current arguments in the section **Better optimization**? in Sec. 3. 3, and I don't think this point is verified empirically as the authors claim. As the authors mention at the end of page 2, the estimator given by Eq. (3)/(4) is consistent, and if Eq. (3)/(4) is minimized perfectly, the obtained model distribution will capture the data distribution exactly. The authors claim that RMwGGIS, with a biased estimator, performs better than ratio matching due to better optimization. Does that mean that the theta learned with the biased objective function (12) actually results in lower value for the objective function in Eq.
(3)/(4)? I. e. somehow optimizing a different objective function (Eq. (12)) leads to better optimization of Eq. (3)/(4) than directly optimizing Eq. (3)/(4)? I would like to see this confirmed in experiments (which is a very simple evaluation). If it's really the case, I would like to see some analysis on why this is the case, since Eq. (12) is picked somewhat randomly, without theoretical justification as Eq. (7). If this is not the case, then there is no better optimization, which means Eq. (3)/(4) is not a good objective function, and the paper should rethink its foundation, or explain where the better performance comes from.
Thank you for this feedback!
Although our current quantitative and qualitative empirical results, as shown in Table 1 and Figure 3 in the revision, can demonstrate that our proposed unbiased and biased RMwGGIS can help to learn better energy functions, we agree that it would be more convincing if it can be shown that **energy functions learned with our methods (Eq. (7) and (12)) actually result in lower value for the objective function in Eq. (4)**. As suggested, we perform a straightforward evaluation to check this.
Specifically, for all the learned energy functions in Table 1, we sample $4000$ data points on each dataset and evaluate the resulting objective value defined by Eq. (4). The results are summarized in the following table. The `1st lowest` and **2nd lowest** are highlighted.
| **Method**| *2spirals* |*8gaussians* |*circles* |*moons* |*pinwheel* |*swissroll* |*checkerboard* |
| ----------- | ----------- | ----------- |----------- |----------- |----------- |----------- |----------- |
| Ratio Matching| 46.02|39.26|31.82|28.57|28.50|37.52|`26.05`|
| **RMwGGIS (unbiased)**| `26.84`|**30.15**|**29.89**|**27.80**|**28.08**|**32.20**|26.09|
| **RMwGGIS (biased)**| **27.15**|`29.31`|`29.54`|`27.75`|`27.30`|`29.45`|**26.06**|
We can observe that our unbiased and biased RMwGGIS indeed achieve lower objective values, which further demonstrates that our proposed RWwGGIS can be optimized better. The results are included in **Appendix E** in our revision.
**More analysis on why this is the case is included in the following 2nd part of our response.**
I thank the authors for the response. The intuitive explanation using hard example mining does make sense, and it's encouraging to see that the biased version indeed achieves better optimization on the original energy function.
But given the current explanation, if the important thing is to achieve better optimization and heuristically we are doing hard example mining, in what sense is the distribution given by Theorem 1 optimal? Would it be possible to have some other estimators that have higher variance but achieves better performance?
In fact the better performance with the biased version already proves the theoretically optimal choice is not necessarily the best, and indicates that the variance of the estimator is not the most important thing here.
The paper identifies a potentially interesting approach to a valid problem, and achieves good performance.
But I do feel the current presentation is a bit confusing and misleading, and there are aspects that require more careful analysis to better understand the improved performance.
Thank you for your response. We are glad that your previous concerns are well addressed. We address the raised specific question below.
> Given the current explanation, if the important thing is to achieve better optimization and heuristically we are doing hard example mining, in what sense is the distribution given by Theorem 1 optimal? Would it be possible to have some other estimators that have higher variance but achieves better performance?
In fact the better performance with the biased version already proves the theoretically optimal choice is not necessarily the best, and indicates that the variance of the estimator is not the most important thing here.
Thank you for proposing this interesting question. The short answer for this question is **Theorem 1 is a must for our unbiased version and the resulting proposal distribution is necessary for our biased version as well.**
We would like to clarify our flow again. We are motivated by the efficiency limitations of the ratio matching method.
Hence, we propose to use Monte Carlo method based on importance sampling to unbiasedly estimate the original ratio matching objective more efficiently. That’s why we propose our unbiased RMwGGIS, as defined in Eq. (7). For our unbiased version, Theorem 1 is a must because it provides an optimal proposal distribution, which yields the minimum variance. This is also verified by our experiments shown in **Figure 3**. In **Figure 3**, RMwRAND is also an unbiased estimator but cannot work due to high variance.
Next, although we are motivated from the efficiency perspective, we observe that our RMwGGIS (Here, we refer to the unbiased version) achieves better performance than the original ratio matching method.
Thus, we conjecture that this is because our RMwGGIS can be optimized better since it focuses on neighbors with low energies, as analyzed in **Section 3. 3**, and performs intuitively like hard negative mining (**Appendix C**). Based on this insight, we further propose our biased version (Eq. (12)) because the biased version also performs hard negative mining (i. e. , focus on neighbors with low energies) and does not include the complicated coefficients (We provide an intuitive explanation on why biased version performs better than the unbiased version in **Appendix D**. The reason is actually the complicated coefficients.
). Note that Theorem 1 is also necessary for our biased version since the proposal distribution of the biased version also follows Theorem 1 (i. e. , the biased and unbiased version use the same proposal distribution). **The coincidence here is that the distribution used for hard negative mining is the same as the optimal importance sampling proposal distribution given by Theorem 1. ** This makes sense because in our case both **importance sampling for Monte Carlo estimation** and **hard negative mining** aim to pay more attention on neighbors with lower energies. Their goals are consistent.
The above explanation clarifies why Theorem 1 is necessary for both our unbiased and biased version. To put it another way, if we do not provide Theorem 1 and gives the biased version (Eq. (12)) directly, It would be confusing and unconvincing since it is a biased estimator and we would lack the detailed derivation for the proposal distribution used in Eq. (12).
**"Would it be possible to have some other estimators that have higher variance but achieves better performance?"** In principle, we can construct other proposal distributions for Eq. (12) to do hard negative sampling. We choose our proposal distribution as Eq. (11) because of the following two points. (1) It can be easily and efficiently obtained by using the gradient information and Tylor approximation. Other proposal distributions don’t have this advantage. (2) The approximated target (i.e., the proposal distribution given by Theorem 1) is sound since it gives higher probabilities for neighbors with lower energies.
**"In fact the better performance with the biased version already proves the theoretically optimal choice is not necessarily the best."** Not really. Note that for biased version, the proposal distribution is also the same distribution (i.e., Eq. (11)) as the unbiased version. Such distribution is our proposed gradient-based approximation for the optimal distribution given by Theorem 1.
Hence, the better performance with the biased version actually shows that the optimal distribution is a good choice.
Thank you again for raising this interesting question. Hope we address it well. Please let us know if you have any further concerns.
I thank the authors for the response.
- In theorem 1 you establish optimality for the estimators as a whole, including the coefficients. The biased version is omitting the coefficients and as a result a different estimator. The fact that the biased version outperforms the unbiased version means that while Theorem 1 can be a consideration (and might be an important one), it shouldn't be the only consideration, as we provably have estimators that are not optimal according to the theory but outperforms the theoretically optimal estimator.
- I get the flow, but I don't like the coincidence part. As mentioned above, I think the variance can be a valuable consideration (and might very well be an important one), but is not the only factor driving the performance. The real factor driving the performance still seems unclear to me, and I believe more analysis/investigation is needed, more specifically from the claimed hard sample mining perspective.
Thanks for the quick response. We do really appreciate it.
**In short, Theorem 1 is only for the unbiased version. And the biased version can be treated as a natural extension of the unbiased version.**
*  **The biased version is a natural extension of the unbiased version.** Our derivation of the biased version from unbiased version is clarified as follows. We firstly connect the better performance of our unbiased version over ratio matching to hard negative mining (Section 3.3 Point (2)).
Then, based on this insight, we propose the biased version by removing the coefficients in the unbiased version because they provide smaller weights for terms with lower energies, which is less effective (Appendix D).
Hence, our biased version is a natural extension of our unbiased version. These two versions are not contradictory with each other. We have made this clearer in our revision (**at the end of Section 3.3, highlighted in brown text**).
* **We noticed that the confusion might be caused by the terminology we used.** The word **“biased”** could sound like to be contradictory to the word **“unbiased”**, thus weakening the consistency relationship between our unbiased version and “biased” version. We would like to replace it with other words, such as **“advanced version”**, if the reviewer suggest so.
* As suggested by the reviewer, we may construct other proposal distributions for our “biased” version. We use our current one since it is efficient to obtain. We can use the gradient information and Tylor approximation to construct it. (Other potential proposal distributions might not have this advantage.)
Thank you and look forward to your reply.
The terminology is fine. My main concern is that from the presented results, it seems like this paper is missing the most important point (hard sample mining). The experiments demonstrate that:
1. Giving up some optimality in terms of estimator variance won't hurt performance (and would in fact improve performance).
2. Moving towards hard sample mining always gives better performance.
At least to me, this suggests hard sample mining should be a more important consideration. Some optimality results from the perspective of hard sample mining, some considerations on efficient computations (already done with the presented approximations) and some more analysis on the better optimization of Eq (4) with hard sample mining would be more valuable than Theorem 1.
The current paper reads like with a series of coincidences we stumble upon a method with good performance that we don't fully understand. I don't find the overall presentation to be convincing.
We heartily appreciate the reviewer for the activate and constructive discussion.
We agree that hard negative mining is a quite important consideration. And in our revision, we do spend a lot of space to discuss the perspective of hard negative mining (as in Section 3.3, Appendix C, and Appendix D.). We also have to point out that Theorem 1 is useful to derive our methodology since it provides a great proposal distribution (from both the perspectives of importance sampling and hard negative mining) and it can be approximated efficiently using the gradient information and Taylor series.
“The current paper reads like with a series of coincidences we stumble upon a method with good performance that we don't fully understand.” We have rewritten our method part (in Section 3.2 and 3.3, especially Section 3.3). We believe it is more convincing now. Please check it.
As suggested by the reviewer, we will think about how to achieve better presentation by balancing the discussion of Theorem 1 and hard negative mining.
At the same time, we also believe that the current draft is quite easy to follow, convincing, and sound (according to the feedback of Reviewer oca8 and Reviewer KK3T), although we use a different writing flow as suggested by the reviewer. Again, we sincerely appreciate the reviewer’s helpful feedback and active discussion. And we hope that the reviewer could reconsider the score penalty associated with the presentation flow. Thank you.
Dear Reviewer Btjs,
Thank you for your previous constructive and active discussion. We have responded to your latest concern about the presentation, and revised our draft accordingly. Since the discussion period will end soon, could you kindly check our response and revision (especially Section 3.3)? Currently, the other three reviewers are all positive about our work. We are looking forward to knowing if we have properly addressed your concerns? Thank you so much.
I have carefully checked the response and revisions, including Section 3.3. My concerns remain. I don't think it's a penalty for presentation flow. In my opinion the current presentation is misleading, and would lead the readers to focus on the wrong points.
I appreciate the added discussions and analysis, but I believe the author should address the concerns in a new version where the main focus  is on the hard sample mining perspective.
Despite the (only marginally) positive reviews from other reviewers, I maintain that this paper is not ready for publication in its current form and would recommend rejection.
We appreciate the reviewer for the response.
*“The current presentation is misleading, and would lead the readers to focus on the wrong points.”* We totally and respectfully disagree with the word “misleading”. **In fact, (1) our current presentation and the perspective of the reviewer are NOT completely contradictory. (2) And we argue that our current presentation is much more clear, reasonable, and convincing.**
(1) We have achieved a consensus with the reviewer that the perspective of hard negative mining is a quite important consideration. It can serve as a reasonable and convincing explanation to support our method. We have indeed described this perspective extensively in Section 3.3, Appendix C, and Appendix D.
(2) The difference between our views is that (2A) we use hard negative mining as a convincing explanation to support our proposed methods, while (2B) the reviewer holds that our method should be completely motivated from the hard negative mining perspective.
(2A) In our current manuscript, our method is motivated from the efficiency perspective of ratio matching, and our proposed method indeed achieves much better efficiency in terms of both time and memory. We observe that our method ALSO achieves better performance, thus we explain this performance improvement using the perspective of hard negative mining, which is considered to be reasonable and convincing by the reviewer (and the other reviewers, such as Reviewer oca8).
Hence, our current presentation is clearly motivated, and the derivation and explanation are sound.
(2B) Differing from our presentation, the reviewer holds that we should focus on the motivation from the perspective of hard negative mining.
However, **this is not a grounded motivation**.
Specifically, we cannot explicitly say that “The performance of ratio matching would be improved if we apply hard negative mining to ratio matching.
Hence we should focus more on the neighbors with low energies.” This is not a grounded motivation. It is just a random guess before we exactly observe performance improvement when applying hard negative mining.
In contrast, in computer vision (specifically, object detection), the motivation of hard negative mining is convincing since we have the PRIOR knowledge that we usually have an overwhelming number of easy negative samples and a small number of hard negative samples [1].
However, we do not have this prior in our case of discrete EBMs, and cannot treat the perspective of hard negative mining as a clear and strong motivation.
To conclude, we argue that the motivation should be problem-oriented. In our manuscript, our method is motivated from the efficiency perspective, which is a well-explained problem of the original ratio matching.
In contrast, hard negative mining is a specific technique to solve or explain the problem, thus it cannot be treated as a strong motivation.
Hence, we believe that our current presentation is much more clear and convincing.
We sincerely thank the reviewer for this discussion. We hope the above response solves your concern. Please let us know if you have further confusion about this point. Thank you.
[1] Shrivastava, Abhinav, Abhinav Gupta, and Ross Girshick. "Training region-based object detectors with online hard example mining." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
Dear Reviewer Btjs,
We respectfully argue that **the motivation should be problem-oriented, instead of result-oriented**. In our manuscript, our method is motivated from the efficiency perspective, which is a well-explained problem of the original ratio matching.
In contrast, you focus on the obtained improved performance (i.e., result-oriented) and hold that we should focus on the motivation from the perspective of hard negative mining, **with ignoring our strong motivation and improvement from the efficiency perspective**. As explained in the last response, focusing on hard negative mining as suggested is not a grounded motivation.
Hence, we believe that our current presentation is much more clear, reasonable, and convincing.
Please let us know if this help us to achieve a better consensus on the presentation. Thank you so much.
To be clear, I acknowledge the motivation and improvement from the efficiency perspective.
That is the reason I'm keeping the score of 5 instead of further lowering it.
While the improvement in efficiency is nice, to me the most surprising bit is actually the better optimization (reviewer oca8 similarly finds this part surprising). Currently this very important point is glossed over. I agree with oca8 that the intuitive explanation is reasonable, but at least to me many questions remain regarding exactly why this is happening, and I believe this point is potentially more important than the efficiency considerations. Current efficiency considerations are grounded in the vanilla ratio matching.
But as experiments show, vanilla ratio matching is far from optimal in terms of performance, and there are many variations that easily outperform vanilla ratio matching (regardless of efficiency considerations).
Finally, I'm not sure why this is not a grounded motivation. This is an interesting and surprising new finding, and you can just state that you find this to be true empirically (some additional theoretical analysis would also be nice). And this is not contradictory to the motivation from efficiency considerations.
Dear Reviewer Btjs,
Thank you for the response. We appreciate you recognizing our motivation and improvement from the efficiency perspective.
(1) *“Many questions remain regarding exactly why better optimization happening.”* This is quite natural and straightforward since our method focus on neighbors with low energies.
Specifically, according to the objective function of ratio matching (i.e., Eq. (4)), neighbors with lower energies contribute more to the objective function (i.e., the loss values for these neighbors are larger). That’s why our methods, focusing on reducing the loss values (i.e., pushing up energies) of these neighbors, indeed lead to lower value for the objective function defined in Eq. (4). This has been verified by the experiment in Appendix E, as suggested by the reviewer.
Hence, it is technically natural and empirically correct that better optimization happens.
(2) *“I'm not sure why this is not a grounded motivation”*. As noted by the reviewer, we have achieved the consensus that both the perspective of hard negative mining and efficiency are quite important considerations. (A) As clarified in [our previous response](https://openreview. net/forum? id=IEKL-OihqX0&noteId=rkJT4twwbd) and agreed by the reviewers (`KK3T`, `oca8`, and even in the initial review of `Btjs`), our current presentation motivated from the efficiency perspective is already clear, reasonable, and convincing. (B) The reviewer suggests that our presentation should be motivated from our empirical observation on hard negative mining (*”you can just state that you find this to be true empirically.
”*). **This is possible but is not as clear/reasonable/convincing as our current presentation. ** As clarified in [our previous response](https://openreview. net/forum? id=IEKL-OihqX0&noteId=rkJT4twwbd), we don’t have an exactly clear and strong intuition/motivation to apply hard negative mining as in objective detection. That’s why we use hard negative mining as a post-hoc explanation instead of a motivation.
On the other hand, we indeed claim that *we find this to be true empirically* in Appendix E. We just didn’t start our presentation with this claim, but we actually present this point and discuss it quite extensively.
To conclude, we cannot convince ourselves to use the suggested motivation based on hard negative mining, given that we already have a convincing motivation and presentation (i.e., the efficiency perspective).
We are happy that we gradually arrive at some consensus with the reviewer. Again, we heartily appreciate the reviewer taking the time to join the discussion. Please let us know if the above responses answer your questions. Thank you.
Dear Reviewer Btjs,
(1) We believe our last response has addressed you concern about "why better optimization happens", from both the intuition side and the empirical side (Appendix E).
(2) We respectfully argue that the suggested motivation and presentation flow are possible but belong to **personal preference**. Again, we don't have a strong motivation to switch to the suggested flow, given that we already have a convincing motivation and presentation, as we detailed in our previous responses.
Since the discussion period will end soon, we hope that **the reviewer could carefully reconsider the significance and the reasonability of our current presentation**. We heartily appreciate the reviewer taking time to actively discuss with us. Thank you!
The paper seems to contain interesting material. Some point must be clarified.
- How to draw samples from your proposal n(x)? this is the main issue in your method.
- Please clarify the relationship with the papers:
V. Elvira, L. Martino, D. Luengo, J. Corander, "A Grandient Adaptive Population Importance Sampler", IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2015.
Ingmar Schuster, Gradient Importance Sampling, arXiv:1507.05781, 2015.
NO.
Thank you for finding our work to be interesting. We make clarifications to the raised issues as follows.
> How to draw samples from your proposal n(x)?
For each training sample $x$, our proposal distribution $n(x_{-i})$, defined in Eq. (11), gives a discrete distribution over $x_{-i}$, for $i=1,2,\cdots,d$. Note that $x_{-i}$ denotes a point in the data space obtained by flipping the $i$-th dimension of $x$.
Then, as shown in Eq (7), we sample $s$ ($s < d$) terms according to the above probability distribution to estimate the original ratio matching objective.
For example, if the data dimension $d=4$ and number of samples we use is $s=2$, for a particular training sample $x$, our proposal distribution $n(x_{-i})$ provides a distribution over $x_{-1}, x_{-2}, x_{-3}$, and $x_{-4}$ (e.g., $n(x_{-1})=0.6$, $n(x_{-2})=0.2$, $n(x_{-3})=0.1$, and $n(x_{-4})=0.1$)).
Afterwards, we sample $s=2$ terms according to the above probability distribution to estimate the original ratio matching objective.
Specifically, in this example, one possible situation of our sampled two terms is $(x_{-1}, x_{-2})$ since they have larger probability according to our proposal distribution.
> Please clarify the relationship with the papers [1] [2].
Thank you for pointing us to these interesting works. Both [1] and [2] use the gradient information of the target distribution to iteratively optimize the proposal distributions for adaptive importance sampling. Such gradient information is widely used in MCMC framework, such as Hamiltonian dynamics, as review in our Section 4. [1] and [2] exploit such gradient information in the importance sampling framework. These two works are significantly different with ours in the following two aspects.
(1) **[1] and [2] are designed for importance sampling over continuous distributions, while our gradient-guided importance sampling is proposed for performing importance sampling over discrete distributions.** Similar to gradient-based MCMC methods, [1] and [2] require the gradient w.r.t. the data space to update the parameters of the proposal distributions (e.g., Eq (4) in [1]).
Thus, they cannot be applied to discrete distribution directly since the resulting parameters (e.g. mean) in each iteration are usually invalid in the discrete data space.
In contrast, we utilize the gradient information to approximate the provable optimal proposal distribution given by our Theorem 1. Our obtained proposal distribution, as shown in Eq (11), is essentially discrete. This point distinguishes our work with [1] and [2] fundamentally.
(2) In addition, in [1] and [2], we have to use iterative update to try to reach the optimal proposal distribution. In our case, we have a provable optimal proposal distribution and then use the gradient information to approximate it in one shot, as shown in Eq (11).
Hence, although we might be able to extend [1] and [2] to discrete distribution in the future, they would be much less efficient compared with our method.
We have included the discussion with [1] and [2] in our revision version (highlighted in red color). Please kindly consider improving the rating score if the above issues are well addressed. Thank you!
[1] V. Elvira, L. Martino, D. Luengo, J. Corander, "A Grandient Adaptive Population Importance Sampler", IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2015.
[2] Ingmar Schuster, Gradient Importance Sampling, arXiv:1507.05781, 2015.
Dear Reviewer DiXS,
We have responded to the raised questions and updated our paper accordingly. Since the time for discussion is limited, could you kindly check our responses and revision? We look forward to your feedback on if we have properly addressed your concerns? Please consider improving the rating score if your questions are addressed. Thank you.
Strengths:
- The paper is well written and contains detailed method derivation and discussions.
Overall, it is easy to follow.
- Nice discussions to related works.
- The method derivation is sound and the proposed solution is a reasonable way to alleviate the expensive computation/memory requirement of the original ratio matching approach.
- Although motivated from computational perspective for fast approximation of an expensive quantity, the proposed method even achieves a better performance than original method in most tasks, which is a surprising result and achieves both computation and performance improvement. The intuitive explanations look reasonable to me.
Weakness/Questions:
- The proposed approach seems to heavily rely on the assumption that the data is binary, which is not applicable to more realistic data such as natural language modeling (or even graph with different types of nodes and edges?).
Indeed, an efficient and effective generalization of ratio matching to non-binary discrete data seems more challenging, interesting and useful.
- The taylor approximation in eq (8) may not be accurate enough since the difference between x and x_-i is not close to zero.
So the approximation error might be larger depending on the structure of the energy function (e.g. non-smooth) and the number of dimension (for d=2, x and x_-i may be very different in the sample space).
- In table 1, there is no standard deviation calculated from multiple random seeds.
Also why biased version is better than unbiased version in most experiments?
- After obtain the energy function, how do you generate graphs? (I suppose using Gibbs sampling?)
- Potential issue with Table 2: I think you are using the same s (number of proposal samples) for various data dimension from 32 to 2048.
However, we typically need larger s for larger dimension d to ensure good performance, and in this case, the speedup/memory saving may not be that large?
- All experiments are "toy" data, possibly because the method is limited to model binary data.
NO.
> In table 1, there is no standard deviation calculated from multiple random seeds.
Also why biased version is better than unbiased version in most experiments?
(1) In Table 1, we report the MMD scalar value to measure the similarity between distributions given by generated samples and real samples, which is widely used to evaluate the performance of generative models. We choose to use $4000$ samples to compute MMD because it is sufficient to obtained stable and convincing MMD.
In other words, if we generate different sets of $4000$ samples and compute the corresponding MMDs, the results are quite stable.
(2) Based on our analysis of the connection between our method and hard negative mining, we provide a possible explanation on why our biased version performs better than unbiased version.
It is obvious that both unbiased version (Eq. (7)) and biased version (Eq. (12)) perform “hard negative sampling”. The difference lies in the coefficients for different terms.
Specifically, the biased version gives the same weights to all sampled terms.
In contrast, the unbiased version provides a weight $\frac{m(x_{-i}^{(t)})}{n(x_{-i}^{(t)})}$ to each sampled term $x_{-i}^{(t)}$, as shown in Eq. (7). Note that $m(x_{-i}^{(t)})=\frac{1}{d}$ for all terms and $n(x_{-i}^{(t)})$ would be larger if $x_{-i}^{(t)}$ has lower energy.
Hence, the unbiased version provides smaller weights for terms with lower energies.
In other words, among its selected offending terms (i.e., hard negative samples), it pay least attention to the most offending terms, which could be less effective than biased version with equal weights. This could explain why our biased RMwGGIS usually performs better than unbiased version.
We have included this explanation in **Appendix D** in our revision.
> After obtain the energy function, how do you generate graphs? (I suppose using Gibbs sampling?)
Yes, we generate graphs using Gibbs sampling on the learned energy function. We have made this clear in our revision.
> Potential issue with Table 2: I think you are using the same s (number of proposal samples) for various data dimension from 32 to 2048.
However, we typically need larger s for larger dimension d to ensure good performance, and in this case, the speedup/memory saving may not be that large?
Thank you for this feedback. Yes, we use the same s=10 for various data dimensions because we observe that $10$ is enough to achieve good energy function for these data dimensions. It is not usually the case that *we typically need larger s for larger dimension d to ensure good performance* due to the potential optimization difficulty.
For example, in our experiment on 256-d dimensional data in Section 5.1, RMwGGIS with s=10 can achieve good performance, while ratio matching with considering all 256 terms cannot, as shown in Figure 4, Appendix F in our revised version. Our Table 2 just gives a rough verification of the efficiency improvements. We agree that the actual improvements should be considered more rigorously.
Thank you again for these constructive and insightful questions! We have revised our paper accordingly. The changes are highlighted in red color (Note that there are changes in Appendix C, D, and E). Please kindly consider improving the rating score if your concerns are addressed.
[1] Hyvärinen, Aapo. "Some extensions of score matching." Computational statistics & data analysis 51.5 (2007): 2499-2512.
[2] Lyu, Siwei. "Interpretation and generalization of score matching." Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence. 2009.
[3] Felzenszwalb, Pedro F., et al. "Object detection with discriminatively trained part-based models." IEEE transactions on pattern analysis and machine intelligence 32.9 (2009): 1627-1645.
[4] Rowley, Henry A., Shumeet Baluja, and Takeo Kanade. "Neural network-based face detection." IEEE Transactions on pattern analysis and machine intelligence 20.1 (1998): 23-38.
[5] Shrivastava, Abhinav, Abhinav Gupta, and Ross Girshick. "Training region-based object detectors with online hard example mining." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
Thank you for your thorough understanding of our work and comprehensive review. We address the raised questions below.
> The proposed approach seems to heavily rely on the assumption that the data is binary, which is not applicable to more realistic data such as natural language modeling (or even graph with different types of nodes and edges?).
Indeed, an efficient and effective generalization of ratio matching to non-binary discrete data seems more challenging, interesting and useful.
> All experiments are "toy" data, possibly because the method is limited to model binary data.
Yes, it is indeed non-trivial to generalize our RMwGGIS efficiently and effectively to general discrete data beyond binary data. Generally speaking, algorithms on discrete data usually starts from considering binary discrete data.
For example, the original ratio matching method [1] is firstly proposed for binary discrete data, and then is generalized to categorical discrete data by [2]. Our current work covers the binary case and would like to consider extending to categorical data in the future.
> The taylor approximation in eq (8) may not be accurate enough since the difference between x and x_-i is not close to zero.
So the approximation error might be larger depending on the structure of the energy function (e.g. non-smooth) and the number of dimension (for d=2, x and x_-i may be very different in the sample space).
Thank you for this interesting question!
(1) Our work is motivated from efficiency (in terms of both time and memory) limitations of ratio matching on **high-dimensional data**. If the number of dimensions is too small, we can use the original ratio matching method directly to learn discrete EBMs since the time and memory requirements in low dimensional cases are affordable. The advantage of our method is that it can achieve both efficiency and performance improvements, compared to ratio matching, in high dimensional cases.
(2) In terms of the Tylor approximation error, we add an analysis in **Appendix C** in our revision.
In addition to an analysis of how our Tylor approximation affects the learning process, we also firstly include a connection between our method and hard negative mining, which provides a new insight about why our RMwGGIS can be optimized better.
The basic idea for hard negative mining [3,4,5] is to pay more attention to hard negative samples during training, which can usually achieve better performance since it can reduce false positives. In our setting of discrete EBMs, each training sample $x$ is a positive sample, and its energy should be pushed down. For each positive sample $x$, all $x_{-i}$ for $i=1,2,\cdots,d$ are negative samples, and their energy should be pushed up. Our RMwGGIS with the specific proposal distribution shown in Eq. (11) can **approximately** choose the $x_{-i}$’s that currently have low energies with larger probabilities. This has the same philosophy as hard negative mining.
Specifically, in our case, $x_{-i}$’s with low energies are hard negative samples since they are the most offending terms, which are close to the positive sample $x$ and have low energies.
The natural follow-up question is **how accurate is the approximation and how does it affect the learning process?** We answer this question to further show that our RMwGGIS is technically sound. Please refer to **Appendix C** in our revised version for a detailed explanation with figure illustrations (in Figure 3).
**More responses are included in the following 2nd part.**
Dear Reviewer oca8,
Thank you for your positive comments and for the interesting questions. We have responded to your raised questions and have updated our manuscript. Could you check our responses and let us know if we have properly addressed your concerns? Please consider improving the rating score if your questions are addressed. Thank you.
Thank you for the response to my feedback. I think it clarified some of my questions and I appreciate the author's effort on updating the draft according to my suggestion. A major concern/limitation - the scalability to more complicated/practical settings such as categorical data still remain as a future work.
So I tend to keep my original rating and I do think this paper is an interesting initial step towards extending ratio matching on learning discrete energy-based model.
A side comment: I think the title is a bit overclaiming - the method only applies to binary data, not any discrete EBMs, and the gap to general discrete EBM seems quite large.
Dear Reviewer oca8,
Thank you for your response and for recognizing our work as an interesting initial step towards extending ratio matching on learning discrete energy-based model. As suggested, we will discuss the limitation/future work in our next revision. We will also consider making the title to be more rigorous. Thank you for your positive comments and constructive suggestions.
Strengths:
Training EBMs on discrete data is a difficult task and few methods currently exist to do so in high dimensions. RM is an appealing option because it avoids the MCMC sampling typically used to train EBMs at scale. The method appears to be a clear win on top of RM (when it can be applied -- not all energy discrete functions can be differentiated).
Beyond this, the method appears correct and valid. My main concerns lie in the method’s ability to scale to higher-dimensional settings than demonstrated in the paper.
Weaknesses:
The main weakness of this work is in its empirical evaluation. The authors only validate the method in two settings; toy datasets and small graph datasets.
Further, their only evaluation is via MMD. Evaluation of EBMs is difficult, but in small-scale settings likelihood can be estimated with AIS. Likelihood-based evaluation would allow comparison to autoregressive models or VAEs, for example.
I am concerned with the comparison to Gibbs-With-Gradients training. How was this done? Did you use PCD? If so, what were your buffer size, refresh rate, and the number of steps per training iteration? PCD can be sensitive to these parameters and if not set properly the results could be very bad. The authors should include these experimental details in the paper (at least in the appendix).
I am not convinced by the scale of the experiments as well. Prior work [1,2] has successfully trained discrete EBMs on datasets with thousands of dimensions. The results in this work are much smaller-scale. Does the method have difficulties scaling to these settings? If so then that is certainly a weakness of the method and it should be mentioned. You could compare results to the MNIST/Potts/Ising training experiments in [1] (no need to do all, but at least one experiment demonstrating scaling ability would be good to have). This would give a better context for the performance of this method relative to other recent approaches.
----------Post Author Response-------------
I thank the authors for their response to my feedback. I appreciate the new experiments. I think they give additional clarity on how the method performs and scales to higher-dimensional data. I am convinced the method has merits and I will raise my score to a 6. I am still concerned that the method will have difficulties scaling to more complicated problems/models and the current results do not demonstrate that.
Having said that, this method is generally applicable and could be a valuable contribution to the community.
NO.
None
[1] Grathwohl, Will, et al. "Oops I Took A Gradient: Scalable Sampling for Discrete Distributions." arXiv preprint arXiv:2102.04509 (2021).
[2] Dai, Hanjun, et al. "Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration." arXiv preprint arXiv:2011.05363 (2020).
Thank you for your constructive review and for recognizing our work to be interesting, novel, and valid. We address the raised concerns as follows.
> The main weakness of this work is in its empirical evaluation. The authors only validate the method in two settings; toy datasets and small graph datasets.
Further, their only evaluation is via MMD. Evaluation of EBMs is difficult, but in small-scale settings likelihood can be estimated with AIS. Likelihood-based evaluation would allow comparison to autoregressive models or VAEs, for example.
Thank you for this comment.
(1) Our existing experiments, i.e., density modeling on synthetic data and graph generation, follow the setting and evaluation metrics of previous works [1,2,3] in the community. Evaluation with MMD is also shown to be valid and consistent in these works.
(2) As suggested below, we add the experiment on training Ising models in **Section 5.3** and **Appdendix I** to show the scaling ability of our method. The details are included in the later response.
> I am concerned with the comparison to Gibbs-With-Gradients training. How was this done? Did you use PCD? If so, what were your buffer size, refresh rate, and the number of steps per training iteration? PCD can be sensitive to these parameters and if not set properly the results could be very bad. The authors should include these experimental details in the paper (at least in the appendix).
Thank you for this feedback.
Yes, we use PCD by following the official implementation of [4]. In the Appendix I.2 of the original GWG paper [4], it is observed that reinitializing buffer samples to noise will result in unstable training and worse performance.
Hence, in our previous experiments, we use buffer reinitializing rate = 0 and tune other sensitive hyperparameters like the number of sampling steps.
Thank you for reminding us that the buffer initialization rate is also important. We re-tune this experiment by tuning the following hyperparameters, and the chosen ones are highlighted: buffer initialization rate (0, 0.2, `0.4`, 0.6) and MCMC steps (100, 200, `500`, 1000, 2000). The results are updated in **Table 3** and the experimental details are provided in **Appendix G**.
> I am not convinced by the scale of the experiments as well. Prior work [1,2] has successfully trained discrete EBMs on datasets with thousands of dimensions. The results in this work are much smaller-scale. Does the method have difficulties scaling to these settings? If so then that is certainly a weakness of the method and it should be mentioned. You could compare results to the MNIST/Potts/Ising training experiments in [1] (no need to do all, but at least one experiment demonstrating scaling ability would be good to have). This would give a better context for the performance of this method relative to other recent approaches.
Thank you for this suggestion. As suggested, we add the experiment on training Ising models to further demonstrate the scaling ability of our method and compare with recent approaches more thoroughly. Please see **Section 5.3** and **Appendix I** of our revision for more details. We summarize the experiment in the following.
Following one setting in [4], we use RMwGGIS to train the Ising model corresponding to a 2D cyclic lattice structure. We create a model using $25 \times 25$ lattice and use the strength of connectivity $\sigma=0.25$, leading to a $625$ dimensional distribution. We set the number of samples $s$ used in our RMwGGIS objective as $10$. We consider EBM (Gibbs) and EBM (GWG) as baselines. The evaluation metrics are the RMSE (between the learned connectivity matrix and the true connectivity matrix) and the running time per iteration. As shown in Tabel 4 in our revision, we achieve a log(RMSE)=-4.00 with an obvious advantage of efficiency. One future direction is to further improve the effectiveness of our RMwGGIS while preserving the efficiency advantage.
Thank you again for raising these helpful questions! We have revised our paper accordingly. The changes are highlighted in red color.
[1] Dai, Hanjun, et al. "Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration." Advances in Neural Information Processing Systems 33 (2020): 10443-10455.
[2] You, Jiaxuan, et al. "Graphrnn: Generating realistic graphs with deep auto-regressive models." International conference on machine learning. PMLR, 2018.
[3] Liu, Jenny, et al. "Graph Normalizing Flows." Advances in Neural Information Processing Systems 32 (2019): 13578-13588.
[4] Grathwohl, Will, et al. "Oops I Took A Gradient: Scalable Sampling for Discrete Distributions." arXiv preprint arXiv:2102.04509 (2021).
Dear Reviewer KK3T,
We made efforts to respond to the initial comments and updated our manuscript accordingly about two weeks ago. Since the discussion period will end soon, could you check our responses and let us know if we have properly addressed your concerns? Any additional questions and comments are warmly welcome.
Also, please consider improving the rating score if your concerns are well addressed. Thank you.
Dear ACs and Reviewers,
We sincerely thank the reviewers for raising the insightful questions and constructive suggestions, which are very helpful for us to strengthen this submission. We have made efforts to respond to the questions and revised our manuscript accordingly. Please kindly check the detailed responses and our revision, where the changes are highlighted with red color.
Besides answering the technical questions,
* We provide an additional insight to understand why our method can be optimized better by connecting it with hard negative sampling (`Reviewer Btjs`).
Further, we analyze how our Tylor approximation affects the learning process (`Reviewer oca8`). Details are included in **Appendix C** in our revision.
* Based on the above insight, we further provide an explanation on why our biased version performs better (`Reviewer oca8`) in **Appendix D**.
* We include one more baseline in graph generation experiments and empirically verify our methods can actually lead to lower value for objective function, as suggested by `Reviewer Btjs`. Details are provided in **Appendix E**.
* We add an experiment on training Ising models to further demonstrate the scaling ability of our method and compare with recent methods more thoroughly, as suggested by `Reviewer KK3T`. The results and details are in **Section 5.3** and **Appendix I**.
* We included the discussion with more related works, as mentioned by `Reviewer DiXS`.
We would like to know if our responses have properly addressed your concerns. Any feedback and/or additional comments are warmly welcomed.

Meta-review:
role: PC
data: comment: The paper introduces a way of making Ratio Matching (RM) scale better to high-dimensional data when training energy-based models (EBMs). The main idea is to estimate the sum over the datapoint dimensions in the RM objective with importance sampling (IS), achieving computational savings by using fewer samples than dimensions. A key part of the method is a proposal that uses gradient information w.r.t. discrete variables to efficiently approximate the optimal (minimum variance) proposal, resulting in much better performance compared to uniform sampling. The authors also introduce a biased version of the estimator that samples from the same proposal but drops the importance weights when averaging over the samples, which, somewhat surprisingly, outperforms the unbiased version.

The idea of using Monte Carlo estimation based on importance sampling to speed up Ratio Matching is novel and sound. The use of gradient information to approximate the optimal IS proposal is also novel in this context, though the idea of using gradients this way to reduce the number of EBM energy function evaluations comes from Grathwohl et al. (2021), where it was used to speed up Gibbs sampling.

While the method is well described, the paper is insufficiently rigorous in several places, most importantly in claiming that Eq. 4 corresponds to Ratio Matching, which is not true. Eq. 4 is instead equivalent to the objective for Generalized Score Matching (GSM) given by Eq. 17 in (Lyu, 2009). Crucially, while both GSM and RM recover the true model if the model class is nonparametric/unconstrained, as is stated in (Lyu, 2009) (and thus agree with each other as well as with maximum likelihood estimation) ), they do not yield the same solution for constrained model classes such as neural networks. This means that the method in the paper implements GSM and not RM. Unlike RM, which has been used widely in the literature, GSM is essentially empirically unproven and thus is a less interesting choice. The main difference between the GSM and RM objectives is the presence of the squashing function g(u) = 1/(1+u) around the probability ratios in RM to avoid division by zero, when the probability in the denominator is vanishingly small (as is explained above Eq. 12 in (Hyvarinen, 2007)). This means GSM is likely to be prone to stability issues due to using probability ratios directly. This is one possible explanation for the puzzling empirical results in the paper, where the proposed sampling-based methods outperform the exact method they are supposed to approximate, with the biased method clearly performing best. The intuition-based arguments made in the paper to explain these results are not convincing and need to be improved upon. While, as the authors pointed out in their response, it is possible to apply the strategy in the paper to RM by applying IS to Eq. 3 instead of Eq. 4, that would be essentially a different paper.

One example of puzzling experimental results is Figure 3, which shows that the base method ("Ratio Matching") does not find the correct solution while the proposed approximate methods do. This suggests that there is something wrong either with the method (e.g. with the objective, as mentioned above) or with the experimental setup. In either case, the cause needs to be thoroughly investigated.

Currently the empirical evaluation is primarily MMD based, relying on sampling from the model using MCMC. Ensuring that MCMC chains mix sufficiently well to sample from the true distribution by visiting all of its modes is difficult, and it is important to provide some evidence that this was done. As suggested by a reviewer, the results would be substantially strengthened by reporting the log-likelihoods for the models, estimated e.g. using AIS, even if that requires including scaled-down versions of some of the experiments.

The title of the paper is misleading and should be changed because the proposed method is specific to EBMs for binary data, even if the intent is to extend it to other types of discrete data in the future.

The clarification and additional results provided by the authors to the reviewers and the AC were appreciated, but unfortunately the outstanding issues with the paper are too major to allow acceptance at this point. The main idea of the paper has substantial promise however, and the authors are encouraged to develop it to its full potential by addressing the points from this meta-review as well as the additional ones from the reviewers.

Bibliography correction: Hyvarinen is the solo author of "Estimation of Non-Normalized Statistical Models by Score Matching". Peter Dayan was the editor of that paper and not a co-author. Please correct your bibliography.

Decision: Reject