Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper explores an orthogonal space gradient descent. This is achieved by constructing the perpendicular space from the gradient direction using $\mathbb{I} -\frac {\nabla g \nabla g^T}{\norm{g}^2} $
strengths_and_weaknesses: The reviewers expertise is limited in this field, hence the comments will be very brief and editorial.
Strengths:
1. The paper is well written. Both the language and the math seem to be consistent.


Weaknesses:
1. The applications might not exactly be suitable for a conference like Neurips
2. The complexity is of the algorithm is not mentioned.
questions: I do not have sufficient context on the paper, therefore I do not have any questions at this point.
limitations: Yes, the authors have addressed the impact of their work sufficiently.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a new variational framework for sampling from constrained domains. By designing an orthogonal-space gradient flow, i.e. the O-Gradient, the proposed methods can sample from manifolds defined by general equality constraints. The O-Gradient can be realized by both the Langevin dynamics and the Stein variational gradient descent, namely the O-Langevin and the O-SVGD. In contrast to previous methods, the O-Langevin and the O-SVGD do not require initializing the samples on the manifolds defined by the constraints. Besides, the paper has proved that the O-Gradient converges to the target distributions with rate $\tilde{\mathcal{O}}(1/T)$, where T is the number of iterations. In the experiment section, the O-SVGD and the O-Langevin have shown their effectiveness in various setups, showing that they can generate samples that approximate the target distribution well while also staying on the manifold defined by the constraints.
strengths_and_weaknesses: Strength:
1. The paper has addressed an interesting topic of variational samplers.
2. The proposed methods have made remarkable progress upon previous methods, alleviating the requirement of initializing samples on the targeted manifold.

Weakness:
1. It seems that the proposed method has some negative impact on the original task, e.g. in the income classification experiment, a decrease in the test accuracy can be observed.
2. It is unclear what the computational cost of this method is compared to the compared baselines.
questions: 1. In Section 4.1, "$\frac{dq}{d\pi}$ and $KL(q||\pi)$ are ill-posed", what is the definition of "ill-posed"?
2. What's the interpretation of Eq. (5)? Why do you choose the function $\psi(x)=\alpha\text{sign}(x)|x|^{1+\beta}$?
3. In line 149, what is the "projection intuition"?
4. Line 252, what is the "energy distance"?
limitations: Yes, the author has adequately addressed the limitations and potential negative societal impact of their work. The limitations of the approach were presented above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a class of algorithms for sampling from a target density conditionally on a set of constraints expressed as an equation $g(x)=0$. The paper relies on a variational formulation of the vector fields defining Langevin dynamics and SVGD to derive an Orthogonal- version of such vector field with two components: an orthogonal component that drives samples toward the set of constraints and a tangential component that diffuses the particles to that they are distributed according to the target. The algorithm is simple to compute and requires only evaluating the score of the target, the constraint g(x), and its gradient. 
Convergence rates for the algorithm are provided and rely on a new stein characterization of conditional measures which appears to generalize stein characterization.
strengths_and_weaknesses: The idea appears to be novel and might have a significant impact in the context of conditional sampling under generic equation constraints. The algorithms are simple to use and appear to be effective on the considered examples. 
The presentation is also clear and the problem well motivated.

 
My main concern is about the soundness of the theory as it appears there are some errors or unclear steps in the proofs (see questions section for more details). I hope the authors can fix these issues and clarify these points. More generally, the write-up of the proofs should be improved to better explain the intermediate steps in the calculations and add explicit logical links between different components of the proofs. 

It seems that the main result relies on at least two strong assumptions: the Poincaré inequality for conditional densities and the inequality in eq (17) on the constraint $g$. Can the authors provide examples where such assumptions hold? Classical PI holds in many situations, but for the conditional case, it is still unclear to me how restrictive this is.



questions: Proof of prop 5.4:
        - It appears the authors apply the k-PI inequality of Definition 5.3 to the function f(y) = \sqrt{q_z(y)/Pi_z(y)}. When computing the variance of f under $\Pi_z$, I get the expression: $1- a^2$ with a = E_{Pi_z}[f], so that k-Pi would be given by $(1-a^2)\leq E[||D\nabla ||^2]$. However, in the proof, the authors use the inequality $2(1-a)\leq E[||D\nabla ||^2]$. I do not see why this would hold, since $1-a^2 \leq 2(1-a) $. 
       - The equation after line 493 suggests that the score difference between $q_z$ and $\Pi_z$ is the limit of the scores $\pi_{\eta,z}$, but I do not see why this is true. It seems this claim follows from Proposition5.2. However, the latter only shows that the densities $\pi_{\eta,z}$ converge towards Pi_z in the 'weak limit'. This result does not necessarily imply that scores of $\pi_{\eta,z}$ converge to the score of $PI_z$ in a pointwise sense. Please clarify this.  
 
- Statement of thm 5.5: I think the statement should be modified as follows:

Set $M_t= sup(|g(x)|, x\in supp (q_0)  )$ and define S_t by dS_t = -psi(S_t) and $S_0=M_0$. If M_0<+infty, then M_t\leq S_t for all times.  Currently, the paper makes the stronger claim that dM_t=-\psi(M_t), but this is not proven and is not really needed for the rest of the paper.
            
In eq 17, there is a factor 2 missing to the 3 terms otherwise the proof in the appendix doesn’t hold.

Following the proof of thm 5.6, it seems that the conclusion of thm 5.6 on the upper bound of the O-stein discrepancy is wrong. Instead, the upper bound should hold only for the min_t over [0,T] and not for any t in [0,T]. 
		
Eq $17$ does not seem to be an appropriate smoothness assumption: when g is smoother the l.h.s in 17 blows up to infinity, so that the constant C_0 must be larger and larger. In the limit case when g is constant, the inequality never holds. This can be fixed by scaling the l.h.s by the norm of $||g||$ and replacing the last term in L 228 by the ratio (\psi(g(x))/\Vert g(x)\Vert). 



limitations: The authors should discuss how restrictive are the assumptions required for the results + fix the proofs. I am willing to raise my score to a (weak) acceptance it these issues are addressed in an acceptable manner.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduced a sampling method, namely O-Gradient, that enables defining equality constraints which is achieved by two-component strategy: decrease the distance to the manifold by decreasing the KL divergence as fast as possible (using a velocity field. The paper presents two formulations using Langevin dynamics (O-Langevin) and SVGD (O-SVGD). The paper presents both theoretical analysis of the convergence of O-Gradient based on Proposition 5.2 that defines a limit that is used to formulate an orthogonal-space Stein equation. This in turn is used to generalize the Fisher divergence. The experimental evaluation considered both synthetic and real scenarios. In the former, the paper presents analysis of convergence comparing O-Gradient with  constrained Langevin dynamics and constrained Hamiltonian Monte Carlo, as baselines, where O-Gradient methods show better convergence when sampling from inside or outside the manifold. The baseline methods have a MAE is almost zero for when sampling is done from inside and are not applicable for sampling from outside the manifold. The real-world experiments are done for income classification, loan classification, and image classification where O-Gradient methods seem to perform better than the alternatives.
strengths_and_weaknesses: In summary, the paper is somewhat clear but more could be added to make it self-contained. The organization is not bad. The originality is mostly about the specific equality constrain considered during sampling. The quality of the work is theoretically good and experimentally reasonable. Some specific details include:

Strengths:
Variational gradient descent applied to sampling with equality constraints is relatively novel problem. It is related to, but not the same than, variational inference subject to inequality constraints for which there are works that propose a solution. 
It is an interesting approach to try to define sampling in terms of closeness and orthogonality to the related manifold defined by the domain of the distribution and the equality constraint. 

Weaknesses:
The paper, as written, is only accessible to the ML sub-community working on Sten variational inference and sampling and related topics as it does not present the intuition of the work but rather in general terms. For instance, the paper states that Fokker–Plank equation of the SDE $dx_t$ (dubbed by the paper as natural guess) does not follow the density flow expected in Eq 10. Further details could clarify this.

There are other sections of the paper where the explanations provided are insufficient such as the lack of a data description in the experiment for Loan Classification with Logic Rules. The function used to measure the test error in the experiment “Prior-Agnostic Bayesian Neural Networks” is not detailed.  
questions: Could you clarify why the FPE of the definition of $dx_t$ in line 150 does not follow Eq (10).
What function is used to compute the CV score in the Income Classification with Fairness Constraint ?
What is the function used to compute the error in Loan Classification with Logic Rules?
What is the function that was used to compute the test error in the experiment “Prior-Agnostic Bayesian Neural Networks”?
limitations: The paper describes the limitations in terms of the nature of the model (i.e., it is an approximation and samples are not exactly on the manifold, the type of constraint is equality only, and continuous time).
No limitations are provided outside this technical realm.
review_title: null

Meta-review: metareview: All reviewers recommend accepting the paper, to various levels of enthusiasm.

When preparing the final version, please take into the following considerations:
- Several of the reviewers pointed out that the paper was unclear/sloppy in places, and that it was not written in a way that is accessible to a general ML audience. Take some time to fix this; people are more likely to read/appreciate/cite/build-on your work if it written in an accessible way, with clear motivation (understandable beyond people in a subfield), and the steps in the analysis are laid clearly in an easy-to-understand way.
confidence: Less certain
Decision: Accept