Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors provide a theoretical comparison between maximum likelihood estimators and score matching estimators in the case where the distributions are from the exponential family.

strength_and_weaknesses: Strengths:
- Proofs are clean and straightforward.
- Theory corroborates with practical observation, that score matching on multimodal data requires noise perturbations.

Weaknesses:
- Theory is mainly descriptive rather than prescriptive --- it is well known that noise-perturbation is crucial for properly learning scores [1].
- Non-asymptotic theory is limited to comparing loss values (KL divergence vs expected square error of scores), which does not provide a clear insight into how the resulting densities themselves compare.
- Asymptotic theory is limited to learning on distributions in the exponential family.

[1] Song, Y. and Ermon, S., 2019. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written, and the proofs are straightforward and easy to follow, even for non-experts in isoperimetric analysis.

To my knowledge, the theoretical connections are novel, though the practical observations they describe have been well-known at the outset of the development of score-based generative models [1].
summary_of_the_review: This work provides a suitable theoretical premise for the well-known observation in score-based generative modeling that noise-perturbations are critical for good performance. For this reason, the most relevant result to me is Theorem 2, which provides theory for this phenomenon under the (albeit simplistic) assumption that the data distribution is exponential.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the statistical properties of the score matching from a geometry viewpoint. The authors show an upper bound for the KL divergence between the underlying data distribution and the score-matching estimator using the log-Sobolev inequality. Moreover, with a focus on the exponential family, they analyzed the asymptotic efficiency of the score-matching estimator compared to the one of the MLE using The Poincaré inequality and isoperimetric inequality.
strength_and_weaknesses: **Strength:**
1. Score matching is a widely used technique and its statistical analysis is of high interest to the community.
1. Their theoretical results seem novel and technically solid to me. I found the isoperimetric viewpoint quite interesting.
1. I found the paper well-written and easy to follow.

**Major comments:**
1. Why is Thm. 3 about an unknown direction? It makes sense to me if the goal is to estimate $||\theta||$ or $w_0^\top \theta$ for a prescribed $w_0$. The fact that the lower bound is proved for an unknown vector $w$ is confusing to me. Is it possible that there exists some $w'$ such that the score-matching estimator achieves better efficiency?
1. The paper could benefit from adding more explanations and discussions. For example,
    1. How large is $C_{LS}(q)$ compared to $C_{LS}(q, \mathcal{P})$ depending on $\mathcal{P}$? Can you give an example?
    1. Can you provide an example for Thm. 2, just like Thms. 1 and 3?
    1. It would be helpful to give some intuition on the tools used in Sec. 2.
1. From (1), $J_p(q) - J_p(p)$ is equal to $1/2$ of the RHS of (6). Prop. 1 seems to be missing a factor of 2. This also holds true in the latter results like Thm. 1.

**Minor comments:**
1. Should $p$ be smooth in Def. 2?
1. What is $f$ in Def. 3?
1. The notation $I_d$ in the paragraph Mollifiers can be confused with the identity matrix.
1. $J_p(q)$ should be $J_p(p)$ in the equation below (6).
1. The subscripts MLE and SM in Def. 6 and Prop. 2 are too large. Maybe use \text{}?
1. In conclusion, "In tihs paper" --> "In this paper".

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. The claims seem to be technically solid and novel. I only have a few concerns which are stated above.
summary_of_the_review: This paper provides useful insights into the popular score-matching estimator. The results seem novel and technically solid to me. I thus vote for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper analyzes the statistical efficiency of the score estimate obtained by the score matching objective and compares it with the maximum likelihood based approach. Specifically, relation between the loss optimized in score matching J has been compared to the KL divergence between the true distribution p and the estimated distribution q. The paper shows that the bound on the KL depends on the log-Sobolev constant, the score matching estimator converges to a distribution that depends on other quantities like Poincare and smoothness of sufficient statistics. Later this experiments support the main idea of this theorem.
strength_and_weaknesses: Many results of the paper are interesting and insightful. And they are fairly novel since this work is probably the first to discuss the statistical efficiency of score estimator.

However, the clarity of the paper could be improved. I have a few suggestions below. Also, with some more insightful experiments, the motivation of the paper could be boosted. 
clarity,_quality,_novelty_and_reproducibility: Generally, the paper is clear and results are of high quality. Also, the statistical efficiency is a new direction in score estimation, so the paper has novel ideas. Perhaps reproducibility could be improved. I have some comments below that might help improve the paper:
I believe there are a couple of things that need to be clarified. First and foremost, while the paper nicely motivates citing the example how new score estimation techniques circumvent many problems associated with probability density estimation and estimation of partition function, their result does not exactly discuss the property of the score function, say s, but rather the estimated distribution q, which is quite different from the score s, i.e. the derivative of log of q. It is unclear at the moment whether these results translate easily to the properties of score s. That is, as is customary in recent score based generative models, score function, s, is modeled as a neural network and minimizes the loss J as defined in eq. 1 of the paper.  Another important and related matter that needs to be clarified in the paper is that the estimation that the paper analyzes is different from the recent developments in score based generative models where to facilitate the estimation of score, distribution is first convolved with a series of Gaussian noise and score is estimated for all the noisy probability distribution indexed by time. While this paper does not discuss this setup, it is better to explicitly clarify this point since score based generative models are quite common and can immediately confuse readers.

A couple of suggestion for the authors to improve the clarity and presentation:
1. The distribution q first mentioned in section 2, what is it and how is it related to the score?? After reading a couple of sentence, I guessed that the score being estimated is the grad of log of q. But, it is better to clarify that relation and introduce q first. Also, this is very critical here. I am still puzzled why analyzing property of q, other than perhaps that is well studied, is relevant in contrast to studying the property of score directly??
2. From the definition of log Sobolev constant in eq. 2, it seems like the log Sobolev constant does depend on the distribution p. But, later in remark 2, authors state that log Sobolev constant does not depend on p, which is what I was expecting. But, it's better to clarify that in Defn 2 and explain why. 

3. The paper is packed with a lot of information, which is hard to unpack for the reader. Perhaps it is better to let go some of the contents and better explain other more relevant ones. For example, how does the score matching estimator in Proposition 2, page 5 come from. Authors have cited the paper by Hyvarinen 2007, but it's better to derive it for clarity.
4. Related to the above comment, pseudolikelihood in section 5 might be a bit less relevant to the analysis of score matching.
5. I found the experiments illuminating and would have liked more of it. Plus, at the moment it only supports theorem 2, something like that for Theorem 1 would have been nice.
6. Lastly,  score estimation in generative models have been linked to log likelihood, which makes me wonder if there is a connection between these results and those. See the following paper:

Chen, T., Liu, G.H. and Theodorou, E.A.. Likelihood Training of Schr\" odinger Bridge using Forward-Backward SDEs Theory. ICLR 2022.

summary_of_the_review: I generally liked the new perspective of analyzing the statistical efficiency of the score matching objective discussed in the paper. Many theoretical results have been presented. However, there were some concerns in presentation and a few clarifying questions. Hence, I am on the positive side.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the statistical efficiency of the score estimator which was still an open question.
The authors relate such efficiency to properties of the target distribution estimated (such as isoperimetry, and Poincaré inequalities).
Given the high interest in using score matching for learning generative models this work provides significant insight on the properties of score matching estimators and when they fail to be statistically efficient. 

All reviewers as well as the AC agree that the paper provides an outstanding theoretical contribution.


justification_for_why_not_lower_score: The paper provides a significant contribution to the understanding of the statistical efficiency of score matching which is of fundamental interest given the widespread use of score matching.
Decision: Accept: notable-top-5%