Sampling  with Trusthworthy Constraints:  A Variational Gradient Framework   
Authors: Xingchao Liu
Xin Tong
qiang liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Trustworthy Machine Learning
Sampling
Bayesian Inference

Abstract: Sampling-based inference and learning techniques, especially Bayesian inference, provide an essential approach to handling uncertainty in machine learning (ML). As these techniques are increasingly used in daily life, it becomes essential to safeguard the ML systems with various trustworthy-related constraints, such as fairness, safety, interpretability. Mathematically, enforcing these constraints in probabilistic inference can be cast into sampling from intractable distributions subject to general nonlinear constraints, for which practical efficient algorithms are still largely missing. In this work, we propose a family of constrained sampling algorithms which generalize Langevin Dynamics (LD) and Stein Variational Gradient Descent (SVGD) to incorporate a moment constraint specified by a general nonlinear function. By exploiting the gradient flow structure of LD and SVGD, we derive two types of algorithms for handling constraints, including a primal-dual gradient approach and the constraint controlled gradient descent approach. We investigate the continuous-time mean-field limit of these algorithms and show that they have O(1/t) convergence under mild conditions. Moreover, the LD variant converges linearly assuming that a log Sobolev like inequality holds. Various numerical experiments are conducted to demonstrate the efficiency of our algorithms in trustworthy settings.

Sampling with Trustworthy Constraints: A Variational Gradient Framework
Xingchao Liu"
Xin T. Tong"
Department of Computer Science University of Texas at Austin xcliu@utexas. edu
Department of Mathematics
National University of Singapore mattxingnus .ww.uuuuuueuuuuuu.e sg
Qiang Liu
Department of Computer Science University of Texas at Austin lqiang8cs. texas. edu
Abstract
Sampling-based inference and learning techniques, especially Bayesian inference, provide an essential approach to handling uncertainty in machine learning (ML). As these techniques are increasingly used daily life, it becomes essential safeguard constraints, such as fairness,
the ML systems with various trustworthy-relater safety, interpretability. Mathematically, enforcing these constraints in probabilistic inference can be cast into sampling from intractable distributions subject to general nonlinear constraints, for which practical efficient algorithms are still largely missing. In this work, we propose a family of constrained sampling algorithms which generalize Langevin Dynamics (LD) and Stein Variational Gradient Descent (SVGD) incorporate a moment constraint specified by general nonlinear function. By exploiting the gradient flow structure LD and SVGD, we derive types of algorithms for handling constraints, including a primal-dual gradient approach and the constraint controlled gradient descent approach. We investigate the continuous-time mean-field limit of these algorithms and show that they have (11/t) convergence under mild conditions. Moreover, the LD variant converges linearly assuming that log Sobolev like inequality holds. Various numerical experiments are conducted to demonstrate the efficiency of our algorithms trustworthy settings.
Introduction
Efficient approximation and sampling methods of intractable distributions plays key role in proba- bilistic machine learning, especially Bayesian inference. Traditionally, this problem has been framed Monte Carlo sampling. Recently, variational optimization ideas, which frame the sampling prob- lem into functional minimization problem KL divergence in the space of distributions, have been popularized, exemplified by Langevin dynamics (through the lens of Wasserstein gradient flow (Jordan et al., 1998)), and Stein variational gradient descent (Liu & Wang, 2016; Liu, 2017; Liu Wang, 2018). Let p be probability density function supported on Rd from which it is intractable draw samples exactly; P5 can be the posterior distribution in Bayesian inference, or probabilistic graphical model constructed based data and expert knowledge. The variational methods consider
*: Equal contribution. Code is available at httppe///gicaaa....ooooooaaaaa/a http:///w.oiii//////raaaaaaaaaaa
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

to minimize the following KL divergence objective in the space of distributions, denoted by P,
min KL II pt).
qEP
which, if solved exactly, recovers the target distribution q = Pg. Langevin dynamics and SVGD can be viewed as solving (1) following two different types of gradient flow under different metrization of P; in practice, both Langevin dynamics and SVGD yield a particle approximation, a set of points (a.k.a. particles) {0: Y C R whose empirical measure q = Σi= 60,/n /n approximates P in weak convergence; the difference is that Langevin dynamics obtains the particles with a random diffusion process, while SVGD evolves the particles with a deterministic repulsive interacting particle system.
However, most existing works focus on sampling from unconstrained domains, reflecting the fact that Problem (1) is an unconstrainted optimization P. In many cases, especially trustworthy machine learning, we would like to impose a constraint of high practical importance in addition to approximating Pū- yielding a constrained variant of (I)
min KT.(q |I P6), 8.l. Ey
'EP
where we find the best approximation of pn subject moment constraint E, [0)] - O, with g a general nonlinear function specified by the users. If pū is the posterior distribution in Bayesian inference, Problem (2) can be viewed as form of posterior regularization (Zhu et al., 2014). Our framework provides powerful tool for enabling reliable trustworthy machine learning with faimess, safety, interpretability, and other constraints, which we illustrate as follows.
Fairness Constraints A motivating example is Bayesian fairness (Chakraborty, 2020; Ji et al., 2020; Dimitrakakis et al., 2019), in which, in addition to approximating the posterior distribution, we want to enforce equalized odds or other fairness constraints on different demographic subgroups.
Specifically, suppose we given dataset D (or(0, ,y(0) (0(0),)()))) Jē=1 consisting of the feature vector at()) a label y()), and protected attribute 20() (e.g., male vs. female), and want to fit it with prediction model ((;;0 0) described by a parameter 0. In typical Bayesian inference, we are interested in finding a distribution g on 0 to approximate the posterior distribution (0) = p(0 (D). In fair Bayesian inference, we also hope to satisfy certain fairness constraints. For example, to control the disparate impact, one can introduce a constraint of Ey]g(()] S with
g(0) E (0) c,
fforr (0) = (covpl2, {z;;)))²
(3)
where we enforce the prediction ((;; 0) to be uncorrelated with the protected attribute z. Therefore, solving (2) allows us to obtain the best approximation of P subject to the faimess constraint.
Safety and Interpretability Constraints It is difficult to control and make sense of the behavior of large AI models such as deep neural networks. In many applications, such as healthcare, robotics, and Al-based systems, it is essential to add guardrails to ensure that the AI models stay within a pre-specified safety region. For example, we may want to ensure that the prediction f(x, 0) must fall inside a pre-specified interval [Jo.- (z). (z)]. which can be ensured by (g)) S 0 with
g(0) = Dist(ū(z;, 0), [úo. (Ir), $0,+ (x))),
where Dist(; .) is a point-to-set distance. Similar approach can also be used to control and increase the interpretability of deep neural networks (DNNs), by enforcing that the prediction of DNNs is close to simple interpretable models such as linear classifiers and logic rules.
Prior-Agnostic Bayesian Inference The posterior of typical Bayesian inference depends on both the likelihood E(O) which represents the fitness on data and prior distribution (let it be ro). A poor choice of prior may heavily influence the posterior and yield poor fitness. One approach to automatically limiting the influence of prior is to find q that minimizes KT.(q po), subject the moment constraint with (0) = (() + e, so that we always have a guaranteed high data fitness (i.c., high E(O) regardless of the choice of the prior. Note that P here is the prior, while it the posterior in the two settings above.
Our Contribution We advance the frontier of sampling with constraints in the following ways: i) We propose two general approaches to extend SVGD and Langevin dynamics the constrained setting (1), including a primal-dual gradient method (see Algorithm 1). and novel constraint controlled gradient descent method (see Algorithm 2). ii) We develop novel theoretical analysis on both methods for both convex and non-convex settings. iii) We demonstrate the power of our approaches on variety of tasks related to trustworthy machine learning, including fair Bayesian classification, incorporating logic rules into black-box models, and training monotonic neural networks

---

Overview of the Main Algorithms
We provide quick summary of the practical algorithms that we develop for solving (1), so that the readers with a practical interest can skip the main theoretical derivation, which in Section 2.1.
Our algorithms iteratively update a set of n particles {0c,) C Rd through iteration 4, such that its empirical distribution, denoted as qt = -i=1 d0:,//n approximately solves the constrained Problem 2 in a proper sense when I. + +00 and n1 + +00. The updates of our algorithm for the SVGD and Langevin cases are:
0i,+11 0i.t + h.B.q. (Vlog (0) A (g(O)) 02.)) + Voke(0, Langevin: 01,+11 = -i,, + l(V logp5(0..)) - Vy((..)) 2hhu.,
SVGD:
(4)
(5)
where {Et,} are i..d. standard Gaussian noise and h is step size. The updates modify the standard SVGD and Langevin update rules by introducing an extra X Vg( term to account the constraint in (2), where A serves Lagrange multipler, whose update rule is described below.
straightforward approach is update X by performing pro- jected gradient descent on the dual problem of (2). As we show in Section 2.2, for both the SVGD and Langevin case, this can be achieved by.
Primal-dual Gradient Method
(6) where h is step size that can different from h. This is similar to the typical primal-dual gradient method on finite dimensional optimization: we increase A-1 1 if E. [0) 2 0 (constraint violated) and decrease it if the constraint is met. See Algorithm I, and the detailed version in Algorithm 3 in the appendix.
A = max + hEs [g))], 0),
Constraint Controlled Gradient Method Because A is iteratively updated in (14), the result can be sensitive to the initialization Ào and the learning rate 17 of In Section 2.3, we propose an alternative "constraint controlled" method so (4) and (5) yield the steepest descent on the KL objective while ensuring that the solution converges to the feasible set rapidly when violated. The derived update X is different for the Langevin and SVGD cases and is shown below:
aF. [g] + E.g. [(V log po) Vg
Vgl
Langevin:
A = max
(7)
Ege
V Vgll"
Cilaang [0)]
Eo,0 Fo V ( (V log + Vo)A:(0,0')
SVGD:
A max
Eg.8' ~ (V (g(0))(((00))
(8)
where a > is a coefficient. Note that the de here is completely decided by the information at the t-th iteration and does not need to be iteratively updated from d like (6). See Algorithm 2 and more details in Algorithm in Appendix.
Main Results
We first introduce the background on SVGD and Langevin through the view of minimizing (1) via functional steepest descent in Section 2.1. We then introduce our two approaches to extending SVGD and Langevin dynamics for solving the constraint optimization in (2) in Section 2.2 and Section 2.3. We provide theoretical analysis of both approaches with and without convexity assumptions.
2.1 Review on Langevin Dynamics and SVGD: Sampling with Steepest Descent
We provide unified introduction to Langevin dynamics and Stein variational gradient descent (SVGD), which can be viewed as minimizing KL(g || P5) with two different types of gradient flow on the space of distributions P. Our results apply to both algorithms. For audience with special interest in one of the two algorithms, we recommend ignoring the other one in the first read.
Assume we start from an initial density 4o and the associated random variable Og 40- We consider moving the random variable 0, along time-dependent vector field or : Rdd - Rdd In other words, 0, is driven by ordinary differential equation (ODE): /dt/dt (e(x)). Let ge be the distribution

---

of 8, at time t. which is known to follow the Fokker-Plandee equation /dt = - (): where fT)()) = -i Og (0) denotes the divergence operator of vector-valued function f: Rd - R with 0f and Se the 8-th element of 0 and respectively; here formally view V as a d-dimensional column vector.
the "T" notation because
To decrease KL II Pt) fast possible, want to choose @t. from pre-decided candidate function set Fs, so that the decreasing rate d KL II pb) is maximized. One can show that
d KL l1 P5) Bo (VV log (0) V log (()) (aa((0)))((00 (.)] == Re
(9)
which is a linear functional Rg. acting on Fe. Assume Fi is Hilbert space, equipped with an inner product (.- and norm 1|+ By Riesz representation theorem, assuming R is continuous linear functional, there exists an element Ege,p, 22 E Fr, such that Ry. [ø] (F Φ) F.. We call r 2. the Riesz representation of R in Fr. The optimal choice of Øt in F, can be framed into optimization:
Φi arg min OEF.
1 { { 6)5, 2 10013
Bs (q:, p5) F 11, (10)
where add a regularization || to constrain the scale of o, yielding the optimal solution d = ; here we defined Dy, (ge: p5)² to be the corresponding maximum descending rate of KL divergence, which can be viewed as a discrepancy measure between qi and P In this work, we always assume that F; is sufficiently large, so that Dr, (gt: p5) = implies q: = pô-
Langevin dynamics and SVGD can be viewed as using different Hilbert spaces Fr, hence yielding different Riesz representation r Tg..P for the linear operator R
Langevin Dynamies Taking F, be Lg. the Hilbert space of E2d Rd maps equipped with inner product (Ø, Φ¹) Lu = Eg (8()) Ø (())). Then it is immediate see from (9) that
(+) = r PG o (0) = V log (0) - V log (().
(11)
In this case, Dr, (gt. p is the Fisher divergence Es-
V log (0) V log q (0)1 Note that V log qe gt)g:) -V (V (0))
the Fokker Planek equation, dg. /dt v ((V log
V (V)(()), coincides with the density functions of the Langevin diffusion process d0, V log p (0.)dt + v2dW,, whose dinereiscettization yields the (unadjusted) Langevin Monte Carlo method; here We is the standard Wiener process. Meanwhile, other discretization methods also exist, see e.g. Wibisono (2018); Salim et al. (2020).
Stein Variational Gradient Descent (SVGD) In SVGD, we take F, to the reproducing kernel Hilbert space of positive definite and continuously differentiable kernel ky (0, 0*). By the reproducing property RKHS and Stein identity (see Liu & Wang (2016)), one can show that the optimal ø, is
(c(+)
Fa. (+) Ea-q. V (0)): (0, Voke }),,
(12)
Related, Day, (qt, p5) reduces to kernel Stein discrepancy (Liu et al., 2016; Chwialkowski et al., 2016):
D (g. = || 1 ||= = Eong. (VV log + Vo) T (V (0")) + V (e 0')
nice property of the Φe in (12), compared with the Φe in (11), is that depends on ge only through the expectation operator Egg. which enables a direct particle implementation of de; = d. instead of resolving to a diffusion process. Specifically, we initialize go = >i ō0../n to be the empirical measure of set of particles {0,,0)s. then qi remains to be an empirical measure, that is. Σm= òø.,/ /n, where the particles are evolved by
d 0r, Ea-qq V log p (O, v..)) + a0k dt
Vi 1,..,,.,.
This is set of differential equations coupled by the mean field qt- Following Liu & Wang (2016). the bandwidth parameter of the kernel (0, 0') at time can depend on the current particles (0c.):: via the median trick.

---

2.2 Primal-Dual Gradient Method
Introducing a Lagrange multipler Problem (2) is equivalent to the following minimax problem:
min max [L(g, A) = KL(g Il pb) + (B))
(13)
pEP A20
We solve the minimax problem by alternatively updating À and q with gradient descent. Note that we can rewrite Lg, A) into L A) = KL(g lI +*)(, with (A)). and (AA) min L(g. A) log Eewpy qEP
px (0) exp(-Ag(
and p = arg gin [g,, X). Therefore, if we want to minimize T(g, )e) with fixed )r we should update dr to move it towards Px, which can done by following the functional gradient descent with either SVGD or Langevin dynamics: d0, /dt = rg..s, (0.). With fixed q, we perform standard projected gradient ascent on shown in (6):
2 = mmax(At-1) + AE. [g [o())), 0),
(14)
where h is a step size. In the continuous time limit (h 0), we find the X follows projected ordinary differential equation. can be written jointly with the density q: of 0g as a primal-dual gradient flow (PDGF):
dg: - (014)) V (Fgors, 9t);
dA, dt
[y lglla
(15)
with [a]x,+ = T(A < 0) max 0) + T(X > 0)a and h/h characterizes the update speed of X relative to qr. Again, Pi, is defined by (11) for the Langevin dynamics, and (12) for the SVGD, as long as pū is replaced by PX,
Numerical Implementation To implement (15) through particle-based algorithm, we use Euler discretization and approximate qu with empirical measure Σi= de,,/nn where {0,.)/=1 are the particles at time f. This yields the updates of SVGD and Langevin shown in (4) and (5). Note that in (5), we run n parallel chains of Langevin dynamics, which are coupled since they share the common A. The empirical distribution of (0a.r} is used to approximate Eg [9() in (14). See Algorithm 1 and more details in Algorithm 3.
Convergence Analysis
One problem of primal-dual gradient method is that the algorithm does not try to meet the constraint directly. Instead, tries to minimize KL(g P which involves the constraint. So in order to prove the convergence, we need to assume that KL(q II pt) and its maximum descending rate =3 pX) dominate the constraint function g. In particular, we consider the following assumptions:
Assumption 2.1. There exist positive constants C1 C2 < co, such that for any t 2 0,
(T [o] - Fps, [g])² S ca DD, (ge, p
(16)
(Egg. [al - Eps, [g]) S C2 || P%.)-
(17)
A Algorithm 2 Constraint Controlled Method
Algorithm 1 Primal-Dual Method
Initialize the particles {0r,0] Let q
Initialize the particles {0 and Ag- Let
Σ, /n.,/n.
qe -e ded.../ / for iteration i do Update the particles {0 by Eq. (4) (SVGD), or Eq. (5) (Langevin).
for iteration & do IrSVG update {0.,) by Eq. (4), and then update X, by Eq. (8)
If Langevin, update {0.,)r Eq. (5), and then Update de by Eq. (7).
Update A by Eq. (14). end for
end for
Condition (17) always holds when g is a bounded function, due to the Pinsker's inequality (See e.g., Lemma A.1 1 of Cui & Tong (2021)). Condition (16) holds if can be written into form of (0) V log (0) T (0) + (T) for some E R, sh E F, and || < (see Lemma B.1 in Appendix). It can also obtained by (17) and log-Sobolev like inequality (18), which we
will discuss later.
5

---

Assumption 2.2. There exists 0 < S ii oo such that do S varomps, la(0)) I for Vt 2 0.
This is mild assumption, and holds when g is bounded and is not constant on the support of px.
Our analysis is based on the following Lyapunov function for the minimax problem on L(q, A):
Flg, x) = L(g, >) - (A) + max (A)-
X
Lemma 2.3. We have Elg, A) 2 0, and Elg. A) = iff (g. X) is a saddle point of Lig, X).
Under mild conditions, we can show E(g, Ae) is decaying along PDGF. Moreover, we can find solution that meets the KKT requirement approximately.
Theorem 2.4. If (16) holds and 0 < 1 < 1/(2),), then E(qt, Xe) decreases aaonotoncccc following the primal-duct gradient flow in (15),
d Elq,, Ne) 2 A(q Me) = (1 2ein) (q:, p2.) (B( I((A or Ey [g] 0).
1
dt
Also (15) finds an approximate solution with an O(1/T) rate: min A(qe. 2
+ Eggo, Ao), VT 2 O.
Note that (qq, X) a measure optimality. This is because s(g, A) 0 implies the KKT condition
of the constrained optimization (1) holds, which is 9 = P2 Eg19l S 0 and 1E = O.
Linear Convergence with Log-Sobolev like Condition We can further show linear convergence
of KI. divergence to the optimal solution by assuming a Logarithmic Sobolev like inequality.
Assumption 2.5. There exists a positive constant C3 < co, such that for any t 2 0,
KL(q P%,) S cgDiy, (gt, p².)
(18)
Theorem 2.6. Suppose problem (13) has solution (q" X²), and Assumptions 2.1,2.2 and 2.5 hold. The PDGF following (15) with < < 1/(2c1) will converge to qº linearly in KL divergence:
KT 11a ") S a1 E(q,, 2e) S on cxpp(oot))(00 2o):
F(go)
where max + c2,
21 + 200 and a2 = min 1 (1 CiT), f m00).
2
Assumption 2.5 is equivalent to the log-Sobolev inequality in the Langevin case (when F; = Cy. and -) is Fisher divergence), which holds when log P is a bounded perturbation of a strongly concave function (Holley & Stroock, 1986). In particular, it holds when P0 is strongly log-coneave and is convex. So it can be seen as a strong convexity (concavity) assumption. In the optimization literature, e.g. Nesterov et al. (2018), is well known such assumption is necessary for linear convergence; without it, we can usually only show O(1/t) convergence similar to Theorem 2.4.
For SVGD, Assumption 2.5 is less well understood. If one replaces g. with any probability measure q. then (18) would fail to hold for kernel Stein discrepancy (KSD) because if qi is a discrete particle but DD,, (gt- px) oo under
mild conditions on the kernel ky; see also Lemma 36 of Duncan et al. (2019). On the other hand, it is unclear (17) will hold for the smaller class of densities where {q: t [0, oo)) takes place in SVGD given that the initialization yo is sufficiently regular (and in what sense). Therefore, Theorem 2.6 is not readily applicable to SVGD. We hope future works draw more understandings on this issue.
measure, we would have KL(g, || p
2.3 Constraint Controlled Gradient Descent
The results of the primal-dual gradient method can be sensitive to the initialization Ån and the learning rate 17 of X. We now propose "constraint controlled" method, which finds constrained variant of steepest descent direction ø that yields the steepest descent on the KL objective like Section 2.1, while ensuring that the solution converges the feasible set rapidly when it is violated.
Following Section 2.1, assume we update qe by d0, = (f. )d,, and want to decide the optimal é. To solve the constrained optimization (2), in addition maximizing the descending rate of KL

---

divergence as Section 2.1, we also want to ensure that the constraint Eg. [g] is properly controlled,
such that 1) if the constraint not met (i.e., 1. [g]
2 0), we should monotonically decrease 2g gg).
and 2) after the constraint is met (i.e., R [g] S 0), we should monotonically descend the loss KT.((t while ensuring that the solution stays within the constraint set. Note that
gg [g] = Ey. (Vg(t) (() == Sq g [ot]] = ($g.g) Fi)F..
dt
where S 2. is linear operator Fi, whose Riesz representation is assumed to be gg.g. Generalizing the functional steepest descent idea in (10), we propose to set Φe to be the solution of
{
dy arg min DEFi
-(rar.p 6)FF + 2 |1bb
s.l.
(S 6) S -alee (g] }.
(19)
where a 2 0 is control coefficient. This ensures that, if the constraint is not met (i.e. Egg [g] > 0), the constraint is descending (az R. [g] S -alE.g, [g] < 0). If the constraint is met (E,g [g] < 0), then allows E,g, [g] to increase, but with a rate smaller than -axlB.y [t). which decreases towards zero when the solution approaches the constraint boundary {q: [6,
0}: this on one hand allows us to have the flexibility to choose & to decrease (L pt)). and on the other hand ensures the solution is confined inside the constraint set. The high-level idea here is similar to the control barrier functions in control theory (e.g., Ames et al., 2019), which works in completely different settings. Note that the constraint in (19) can be viewed as a linearization of the constant Egg [g] S 0 around qt-
as a: a. [g] gg [g] + & F. [g] axEnng [g] (S...g 5) while the loss in (19) can be viewed as a simple quadratic approximation of KL(g || p5). Therefore, (19) can be viewed as functional variant of sequential quadratic programming (Nocedal & Wright, 2006).
Using the Lagrange duality of (19), it is easy to derive that de Fgr.pō A1Sg.. Tg. with
C.lBeg [g]
+ ( (Far.ng Sq.a)) Fi
0) 0).
A max
(20)
|| 1F
See Lemma B.2 in Appendix. Therefore, the density following the constraint controlled gradient flow (CCGF) should satisfy:
dg: - (dig)) = - (Ig.P5, 9t)
(21)
dt
This is similar to PDGF (15) as is driven towards P3.- But the A, in PDGF is recursively updated by the algorithm, while CCGF has the explicit formula for A Intuitively, CCGF will be more efficient.
Next, we obtain Langevin and SVGD variants by taking Fe to be C and RKHS, respectively.
Langevin Dynamics and its implementation Assume Fe = L. We have Tg..p = V log p v logg qe and S,.9 Vg. Hence d; V log Pa Vg Vlog qt: with A, defined in (7). The evolution of associated with dØ = oe (f) is standard Fokker Planck equation associated with PX. which can be realized by the same parallel Langevin particle dynamics in (5), when setting l: = Ei 8t,/n. Note that X depends on g. only through the expectation E,, [.], which can be replaced by the empirical average of the particles. See Algorithm 2 and more details in Algorithm 4.
The SVGD Case Let F, be the RKHS of kernel ky (0, 0'). Then it is easy to show that Sq..0 (.)
Eg (eg (e (0, -)]. Therefore, (x(.) = Egg [(V log P5 (0) - X (g)0)),,
V (0, }],
with de defined in (8). Similar to regular SVGD, we set g: Σ ōg../n, which is iteratively updated with SVGD updates with P\, as the target distribution as (4). Also, Xe again only depends
on E,, [-], which should be evaluated with the empirical mean of the particles, See Algorithm 2 and the detailed version Algorithm 4.
Convergence Analysis
Since CCGF tries to meet the constraint explicitly, we have the following attractive properties
Theorem 2.7. Suppose || + O in the CCGF (21). Then with &r defined in (20), we have
Bol exp(-ot)EEm)) [g], Vt 2 0.
If Ey,, [9] < for some to 2 0, then Eg [9] S 0 and d KL( || p5) S 0 for all t 2 s.
7

---

Note that the requirement و| # O is necessary for the X in CCGF to be well defined. If fact, if || g1ls, 0, then d F. [g] 0 along any update direction Φ. And if Fi f., one can show di g is a constant function a.a.s. In other words, this is an ill-posed scenario where local methods like CCGF have no chance to solve. An assumption slightly stronger than X + co would be assuming it is bounded from above:
Assumption 2.8. There exists an upper bound Amax,+ < 0o, such that for all time when the constraint is not satisfied (i.e., Ey [9] > 0), we have x S Amax,+ in (20).
This is mild regularity condition, which holds, for example, if Eg [9] and || are upper bound, and ||s...|| > C_ > In the following, we show that, under Assumption 2.8, our algorithm meets the KKT condition with (1/tt) rate. Further, if Assumption 2.5 holds, then (L || q²") converges to zero exponentially fast. Note that the conditions here are milder than that of the primal-dual method (which needs Assumption 2.1 and 2.2).
Theorem 2.9. Let }go} be the density function of d0,- (O)dl with de defined in (20). Under Assumption 2.8, we have
1 min A2 Ar) = Dy (g:, Ph.)² -g [gl)+ (KL(gu. Pō) + Amax, (B [9])+). (E(O,T]
Note that, given that Eyg. [g] S (see Theorem 2.7), A2(9. Ad) implies the rest of KKT condition (i.e., a: = Pt, and (-Xe [g])+ = 0).
We establish below the linear convergence of KL divergence under the log- Sobolev condition (18).
Theorem 2.10. Suppose problem (13) has solution (q", >²), ||8q.. * O, and Assumption 2.5 also holds. Then the CCGF following (21) with a - 1/03 will converge to q" linearly in KL. divergence:
KL(gi !1 S <e q"), Vt [0, +00).
Similar to Theorem 2.6, this result is not readily applicable to the SVGD case because (18) can not be verified for SVGD.
Experiments
Algorithms and Settings We summarize the tested methods and the unconstrained baselines here: Langevin: Vanilla Langevin dynamics. We n parallel chains of Langevin dynamics for sampling; SVGD: Vanilla SVGD with codes from Liu & Wang (2016); Primal -ual--Daa---anieinn Langevin dynamics with Primal-dual gradient descent;
SVGD with Primal- dual gradient descent; Control +Lageevin Langevin dynamics with constraint controlled gradient descent; Control+SVGD. SVGD with constraint controlled gradient descent. For the SVGD methods, we follow the configuration in Liu & Wang (2016). We use the RBF kernel with bandwidth chosen by the standard median trick, that is, use (0, 0') exp(- |0 0'112 /w?)) where the bandwidth set by wi Median/ ||0,, 03.eIl * j} based on the particles {0.,) )=1 at the /-th iteration.
As remark regarding the choice of kernel, Gorham & Mackey (2017) provided a counter-example that suggests that Stein discrepancy with Gaussian RBF kernel may fail metrize the weak con- vergence and suggested to use inverse multi qulti-quadrraic (IMQ) kernel which does not suffer from the problem. However, this counter-example assumes use fixed bandwidth and does not hold for Gaussian RBF kernel equipped with the median trick which can adapt to the scale of the data better. We leave the study of better and adaptive choice of kernels future works. In our implementation. we adopt the same decaying step size as suggested in Welling & Teh (201 1) for both SVGD and Langevin dynamics, where ht, ho (1.0 + -0.55 and ho a hyper-parameter. The step size for the Lagrangian multiplier in primal-dual methods is chosen from (0.001, 0.01, 0.1, 10. 100, 1000). The pyram--arameters are determined by grid-search to reach the smallest constraint loss in each experiment.
8

---

Primal- Dual Langevin
Control+Langevin
SVGD
642
Logie Loss
0.31
Test Violation
Mono Loss
Text Mano Loes
0.32
0.0
0332
334
13.55
-397
Training
Test Accuracy
Training LL
Test LL
(a) Logic Constraint on Loan Application
(b) Monotonic Constraint on COMPAS
Figure 1: Trade-off curve with different 'LL': log-likelihooke 'Mono': monotonicity, Note that the
x-axis are flipped so larger values are on the left.
Lrimae-
Primal-Dual+
Control+Langevin CoNDrol+SVGD
Disparate Impaet
Faimess Loss
0.04
Training LL
Fairness
8.05
-60
1306
Training
Accuracy
Iterations
Tterations
(a) Trfde-off Curve with Different
(b) Training Curve
Figure 2: Experiment results on learning fair Bayesian neural networks. 'LL': log- likelihhooddd Note
that in (a) the x-axis are flipped so larger values are on the left.
Embedding Logic Rules into Black-box Models ML models can provide accurate predictions but are difficult interpret and control explicitly. Our method can be used to enforce the ML model to be consistent with a set of logic rules to improve the interpretability and safety. We illustrate this with loan classification problem of predicting whether lend loans applicant. We impose the two logic rules: (1) an applicant must be denied if she has the lowest credit rank and not employed; (2) applicant must be approved if she has the highest credit rank and has been employed over 15 years. To apply our method, we set PI to be the typical posterior distribution of Bayesian logistic regression, and define the constraint g(0) Liogie (0) - e, with loggie (0) ([ (e.y))>ioogu [Loss(y: ((x; 0))]. where Dingie is the uniform distribution on the (1r, y) that satisfy the logic constraints, Loss is the classification loss and ((;; 0) is the prediction by the Bayesian logistic regression model. We vary the threshold in {0.001, 0.01, 0.02, 0.03, 0.04, 0.05) and find the corresponding training log-likelihood (LL). In combination, they are plotted as a trade-off' curve shown in Figure (a. left). In Figure I(a, right), we plot the trade-off curve of the testing accuracy vs. the constraint violation ((Egngl [flogic()) c)+) )) on the testing data. We can see that the controlled- beaseeled-bbsedd methods tend to enforce the constraints better (reaching lower values in logic loss) in both training and testing set, and the SVGD methods tend to achieve higher test accuracy.
specific
Training Monotonic Bayesian Neural Networks In some applications, it can be desirable to enforce the MI prediction to be monotonic w.r.t certain attributes (Karpf, 1991; Sill, 1998). For example, when predicting admission decisions, fair MI system must admit student with higher GPA over the students with lower GPA, given that they are identical in the other features. We apply our method to enforce monotonicity in Bayesian neural networks. We use the COMPAS dataset following the setting in Liu et al. (2020). In this case, Pū is the posterior of Bayesian neural network on the data, and g is defined as g(0)= = mmon (0) with mono (0) EgwD - ((x; +))+ Here, Emoro denotes the subset of features to which the output should be monotonic. By varying in {0.0001, 0.01, 0.1, 0.5). we plot in Figure ((b) the trade-off curve of LL vs. the monotonic loss on training/testing set, which follow similar trends as Figure ((a).
Training Fair Bayesian Neural Networks We use the constraint defined in (3), and use the Adul Income dataset (Kohavi, 1996), which is a classification problem of predicting whether the annual income of person is 2 $50,000, with gender as the protected attribute. For the experiment, we follow the setting in Martinez et al. (2020); Liu & Vicente (2020). With = {0.0001, 0.001, 0.005, 0.01 }. Figure 2(a, left) shows the trade-off curve of training LL vs. fairness loss, and Figure 2(a. right) shows the testing accuracy vs. CV score (Calders & Verwer, 2010), standard measure of disparate impact. Figure 2(b) shows an example of training curves. We can again see that the control based methods enforce the constraints more strictly and in faster speed.

---

Constrained Sampling on Generative Models We ap- ply our method on Noise Conditional Score Networks (NCSN) (Song & Ermon, 2019). NCSN models the dis- tribution Pó of images by training a neural network to estimate its score function Vlog then generates im- ages by running Langevin dynamics with the learned score function. Here, we add constraints to draw rare samples from the generative model. In this experiment constrain the center of the generated images to be black square, by defining the constraint function to be g(0) | c, where 8 denotes the whole image, 8g is a image patch on region S2 which is the square at the center of image. We use threshold of e 0.0001. We use Control+Langer straint, the original NCSN model is based on Langevin dynamics. As shown in Fig. 3, our method successfully generates diverse images without violating the constraints.
(a) Uncunstrained Samples
(b) Control Langevin
Figure 3: (a): images sampled from the original NCSN model (Song & Ermon, 2019) on CIFARIO. (b): constrained im- ages sampled by Control+Langevin using the score function learned by NCSN, where we constrain the center of the generated images to be black square. Our method generates a variety of shadows following the constraint.
to draw samples under the con-
Related Works, Limitations, Conclusion
Primal-Dual vs. Constraint-Control This work has proposed two methods, primal-dual and CCGF. These two methods are derived from two different perspectives. CCGF tries to fit the constraints directly, while the primal-dual operates in a more indirect manner. This is reflected in the proof techniques, as we can directly show the constraint decays linearly for the CCGF method, but we can only control the constrain through a Lyapunov function with the primal dual method. We believe that the conditions required by CCGF is intrinsically weaker than that of primal-dual. This can also be told from the fact that primal-dual method's d's performance depends on the initialization of Xe and the step size 17. But the CCGF method does not have this issue.
Related Works Although sampling on unconstrained domain has been extensively studied, efficient algorithms for constrained settings are largely lacking. A body of works have developed to extend Langevin dynamics and SVGD on Riemannian manifolds (Girolami & Calderhead, 2011; Patterson & Teh, 2013; Liu & Zhu, 2018); these methods, however, rely on computationally efficient characteri- zation of the manifolds and as a result does not work efficiently with constraints specified by general nonlinear inequalities. Projection-based methods (e.g. Sen et al., 2018) can also be developed for sampling on constrained spaces, but again only work for simple constraints with efficient projection maps. Note that our method considers different setting when the constraint is applied the population (rather than individual particles) and works for general nonlinear inequality constraints. In terms of problem formulation, our constrained optimization in (2) coincides with the posterior regularization framework of Zhu et al. (2014); but we focus on particle- based inference based on SVGD and Langevin, which was not studied before.
Limits, Impacts, Future Directions Our new methods cast full spectrum of theoretical questions that can not be completed with single work. In the current work, we mainly focus on the continuous time and the infinite particle limit n - +000 when ge is smooth probability density. natural future direction is to study the guarantees with finite particle size and discrete time. Another valuable direction is to study whether and how linear convergence can be established for SVGD; see Duncan et al. (2019) for an in-depth discussion of this issue for unconstrained SVGD.
On the practical side, since our methods leverage SVGD and Langevin dynamics, they necessarily inherent their limits, such as the sensitivity on the choice of kernel and step sizes. Another practical issue is that we can only enforce constraints on the training set, and hence generalization error need to be considered if we need to strictly enforce constraints on the testing set. Also, this work focuses on a single constraint for simplicity, which sufficient for most practical applications since different constraints can be combined into single constraint easily. Extensions to multiple constraints are straightforward but will be studied separately.
In terms of social impact, our method aims to impose trustworthy constraints in Bayesian sampling, and can help increase the reliability, fairness, and interpretability of ML systems applied to daily life. We will release code freely online to promote the applications of our approach, although it does open the possibility of malicious use of our techniques for adversarial purposes.
10

---

Acknowledgements
The work is conducted in part in the statistical learning and Al group in computer science at UT Austin, which is supported in part by CAREER-1846421, SenSE-2037267, EAGER-2041277, and Office of Navy Research, and NSF AI Institute for Foundations of Machine Learning (lFML). Xingchao Liu is supported in part by a funding from BP. X. T. Tong's s research is supported by Singapore MOE Academic Research Funds R-146-000-292-114 The authors thank the reviewers for all the suggestions made in the reviewing process.
References
Ames, A. D., Coogan, S., Egerstedt, M., Notomista, G., Sreenath, K., and Tabuada, P. Control barrier functions: Theory and applications. 2019 18th European Control Conference (ECC), pp- 3420-3431. IEEE, 2019.
Calders, T. and Verwer, S. Three naive bayes approaches for discrimination-free classification. Data Mining and Knowledge Discovery, 21(2):277-292, 2010.
Chakraborty, I. Bayesian sampler: Fairness in sampling. 2020 19th IEEE International Conference Machine Learning and Applications (ICMLA), pp. 246-251. IEEE, 2020.
Chwialkowski, K., Strathmann, H., and Gretton, A. A kernel test of goodness of fit. International Conference on Machine Learning, 2016.
Cui, T. and Tong. X. T. A unified performance analysis of tikelihood-infomee subspace methods. arXiv preprint arXiv:2101.02411 2021.
Dimitrakakis, C., Liu, Y., Parkes, D. C., and Radanovic, G. Bayesian faimesss. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 509-516 2019.
Duncan, A., Nüsken, N., and Szpruch, L. On the geometry of stein variational gradient descent. arXiv 2019.
preprint
Girolami, M. and Calderhead, B. Riemann manifold langevin and hamiltonian monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 1332)1123-214, 2011.
Gorham, J. and Mackey, L. Measuring sample quality with kernels. In International Conference Machine Learning, pp. 1292-1301. PMLR, 2017."
Holley, R. and Stroock, D. W. Logarithmic sobolev inequalities and stochastic ising models. 1986.
Angwin, J. Larson, M. and Kirchner, Machine bias: There's software used across the country to predict future criminals. and it's biased against blacks. ProPublica, 2016.
Ji, D., Smyth, P., and Steyvers, M. Can i trust my fairness metric? assessing fairness with unlabeled data and bayesian inference. arXiv preprint arXiv:2010.09851, 200.
Jordan, R., Kinderlchrer, D., and Otto, F. The variational formulation of the Fokker-Planck equation. SIAM journal on mathematical analysis, 29(1), 1998.
Karpf, J. Inductive modelling in law: example based expert systems in administrative law. In Proceedings of the 3rd international conference on Artificial intelligence and law, pp. 297-306, 1991.
Kohavi, R. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In Kdd, volume 96, pp. 202-207, 1996.
Liu, C. and Zhu, J. Riemannian stein variational gradient descent for bayesian inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Liu, Q Stein variational gradient descent as gradient flow. Advances Neural Information Processing Systems (NeurIPS), 2017.

---

Liu. Q. and Wang, D. Stein variational gradient descent: A general purpose Bayesian inference algorithm. Advances in Neural Information Processing Systems (NeurIPS), 2016.
Liu, Q and Wang, D. Stein variational gradient descent as moment matching. Advances Neural Information Processing Systems (NeurIPS), 2018.
Liu, Q.. Lee, J. D., and Jordan, M. A kernelized Stein discrepancy for goodness-of-ft tests and model evaluation. In International Conference on Machine Learning, 2016.
Liu, S. and Vicente, L. N. Accuracy and fairness trade-offs in machine learning: A stochastic multi-objective approach. arXiv preprint arXiv:2008.01132. 2020.
Liu, X., Han, X., Zhang, N., and Liu, Q. Certified monotonic neural networks. Advances in Neural Information Processing Systems (NeurIPS). 2020.
Martinez, N., Bertran, M., and Sapiro, G. Minimax pareto faimesss: A multi objective perspective. In International Conference on Machine Learning, pp. 6753-6764. PMLR, 2020.
Nesterov, Y. et al. Lectures on convex optimization, volume 137. Springer, 2018
Nocedal, J. and Wright, S. Numerical optimization. Springer Science & Business Media, 2006.
Patterson, S. and Teh, Y. W. Stochastic gradient riemannian langevin dynamics on the probability simplex. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Salim, A., Korba, A., and Luise, The wasserstein proximal gradient algorithm. arXiv preprint 0030: 2020.
Sen, D., Patra. S.. and Dunson, D. Constrained inference through posterior projections. arXiv preprint arXiv:1812.05741, 2018.
Sill, J. Monotonic networks. Advances in neural information processing systems, pp. 661-667, 1998.
Song. Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11895-11907, 2019.
Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient Langevin dynamics. In International Conference on Machine Learning, 2011.
Wibisono, A. Sampling as optimization in the space of measures: The langevin dynamics as a composite optimization problem. In Conference Learning Theory, pp. 2093-3027. PMLR, 2018.
Zhu, J., Chen, N., and Xing, E. P. Bayesian inference with posterior regularization and applications to infinite latent syms. The Journal of Machine Learning Research, 15(1)1179-11847, 2014.
12


Reviews:
[{"review_id": "uBFSTeC5ptH", "reviewer": "Reviewer_HwFo", "summary": "The paper presents SVGD variants that can handle constraints on the optimized distribution in the form of an expectation under it. The methods are based on Lagrangian multiplier (primal-dual form) and constraint controlled optimization, respectively. Their convergence analyses in the context of a constraint are developed. Experiment verifies a good fit to the target distribution with the satisfaction of the constraint.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Under a critical view, the considered task seems not to require fancy techniques. The primal-dual form transforms the target distribution to satisfy the constraint, and the constraint control version enforces the constraint to hold in every optimization step (similar to projection in each step). So I did not feel lighted on its originality. Nevertheless, the paper has done this in some fine details, including convergence analyses that take the constraint into consideration, and a good introduction and application results on tasks that require such a constraint.\n\nOne technical question: The authors mentioned that the conditions of CCGF is milder than the primal-dual version. Does that mean that CCGF indeed has a better convergence guarantee in more scenarios, or is it just a proving technique issue?\n\nAlthough the organization is logical and intuitive explanations of math are provided, the writing of the paper can be further improved. To name a few:\n  - Line 2, \"to handling\". Line 21, \"which frame\". Three lines above Line 106, \"to be an\". Line 112, missing \"(\". Line 144, \"holds\". Line 153, \"$E$\" but not \"\\mathbb{E}\". Line 166, \"direction\". Second line in Page 6, \"in addition\". Line 174, \"allows us to have\". Line 202, \"defined in (17), we\". Line 225, \"trustworthy\".\n  - Line 130: \"The empirical distribution of ... is used to approximate ... in (19)\": I suppose the samples are $\\{\\theta_i\\}$ but not $\\{\\xi_i\\}$, and Eq. (9) but not (19).\n  - In Eq. (10), what is $\\eta$? A step size parameter is not expected in such a continuous-time equation.\n  - Eq. (14): $D(q_t, p^*_{\\lambda_t})$ but not $D(q_t \\Vert p^*_{\\lambda_t})$.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "jiH6CtQ8mKU", "reviewer": "Reviewer_mFWB", "summary": "The authors propose two ways to extend Unadjusted Langevin Algorithm (ULA) and Stein Variational Gradient Descent (SVGD) for sampling from a distribution known up to its normalization constant, under moment constraints. The first one is a primal-dual gradient descent scheme,\nwhile the second one is a controlled gradient descent one.\n\nThey provide convergence guarantees in continuous time for both schemes: a 1/T rate for a given optimality criterion Delta in the more generic case (Th 3.4 and 3.8), and linear convergence rate in KL if some form of log-concavity is satisfied for the current target p_{lamba_t}* (Th 3.5 and 3.9).\n\nThese methods perform reasonably well on synthetic and real world experiments, with a better performance for controlled-based methods.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles the interesting problem of sampling under moment constraints. I am quite convinced about the importance of this problem, as motivated in the introduction or the experiments conducted (e.g. bayesian logistic regression with logic rules, or training fair bayesian neural networks).\n\nThey proposed two extensions of ULA and SVGD which are widely used in Bayesian inference, based on schemes from the optimization literature.\nThese ideas are natural but are novel up in sampling to the best of my knowledge. Also, the authors did a good job in providing both theoretical guarantees and empirical evidence for the relevance of these schemes for sampling.\n\nMy main concern is regarding the theoretical guarantees.\nRegardig the first scheme, assumption 3.1 is problematic, as it has to be satisfied at all times for the current target distribution p_lambda_t *.\nFor instance, eq 14 corresponds to assuming the log-Sobolev inequality for p_lambda_t* when F_t=L2(q_t)/in the Langevin setting (satisfied for instance when p_lambda_t* is c_2 strongly log concave) as explained by the authors, or to the Stein log Sobolev inequality when F_t= RKHS /in the SVGD setting, which is hardly satisfied (see Lemma 36 in “on the geometry of Stein variational gradient descent” , Duncan, Nusken, 2019 - does not hold for a bounded kernel such as gaussian or IMQ if p_lambda_t* has exponential tails).\n\nBut maybe it is common in the euclidean optimization framework to require such assumptions on the current iterates to prove convergence for primal-dual gradient methods. Can the authors point to such references?I have a similar concern for the second scheme, and the requirement that the norm of s_{q_t, g} is non zero or lower bounded by c_ . \n\nStill, Theorem 3.5  and 3.9 are mainly valid for the Langevin setting, but are not likely to be satisfied for SVGD -> this should be written clearly.\n\nOverall, I tend towards a weak accept.\n\n\nMinor comments:\n\nSection 2: \n\nWhen F_t is a RKHS, D is known to metrize convergence between q_t and p_0* only under specific conditions on p_0* and the reproducing kernel, see “Measuring sample quality” Gorham and Mackey, 2017. In particular it may fail for a gaussian kernel, which is widely used in practice. This is worth mentioning in the paper.\n\nl101: whose time-discretisation -> one of the time discretisation leads to Unadjusted Langevin Monte Carlo algorithm. Many other time-discretizations of this flow exist, see  “Sampling as optimisation over the space of measures”, A. Wibisono, Colt 2018; or “Wasserstein Proximal Gradient” A. Salim, A. Korba, G. Luise, Neurips 2020.\n\nSection 3:\n\neq 10: what is eta?\neq 11 and 12: p*-> po*?\nl189 : grad log q -> grad loq po*\nl196: same with p\n\nAssumption 3.7 is not written entirely…\nFigure 1: I would have been curious about the non-constrained baseline (ie classical langevin or svgd) to have a better idea of the attained loss and E_q[g(\\theta)]\n\nSection 4 :\n\nl256 : i have troubles understanding how the distribution D_logic is created. For the points x that do not satisfy the logic constraint, it looks like this constraint penalizes the original KL objective as well; but maybe i’m getting it wrong.\n\nWhat is the fairness loss and disparate impact? Why don’t you plot the (average) prediction loss of the BNN and the constraint violation as in the first experiments?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations : cf comments, e.g. regarding theoretical guarantees and SVGD.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "dfofz1sOvE", "reviewer": "Reviewer_1Fbx", "summary": "This paper proposes to modify existing approximation posterior inference algorithms to satisfy an additional moment constraint that might be related to trustworthiness, fairness, or something else. Two algorithms are proposed, which can be used with either a discretized Langevin algorithm or Stein variational gradient descent. Convergence properties of the underlying gradient flows are also characterized. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Overall the work seems quite original and creative. The moment constraint is introduced by extending the usual KL minimizing gradient flow problem as a minimax problem. This leads to a rather elegant “primal-dual” gradient flow formulation, which is discretized to obtain the first proposed algorithm. The second method is obtained as a trust-region–type method using a linearization of the constraint. In addition to proposing two algorithms, convergence results for the (continuous-time, infinite-particle limit) of both algorithms are provided. \n\nQuality: The paper appears to be technically sound. However, the results concerning the SVGD flow are misleading. This is because most of the theory requires a log Sobolev inequality to hold. However, it appears that an LSI essentially never holds for SVGD [DNS19]. Thus, the results as they apply to SVGD are essentially vacuous. Moreover, we should not expect SVGD to behave well for multimodal distributions, since the KSD does a poor job detecting the relative mass of different mods. Given all this, it is unsurprising that the Langevin algorithms are empirically superior. But the superiority of the Langevin algorithms is never clearly acknowledged. If anything, the author(s) seem favorably disposed to the SVGD approach, despite its clear theoretical and empirical inferiority. \n\nClarity: While the motivation of the paper is clear, the theoretical results are extremely dense and hard to digest – particularly for a reader who isn’t already familiar with the relevant background material on gradient flows. In light of the better performance of the Langevin-based algorithms, to improve readability I would suggest moving all of the technical SVGD content and the general framework to the appendix and focus on the more digestible Langevin case. \n\nSignificance: The results and approaches developed seem to be potentially quite useful and could open up a number of interesting new avenues for future work. Thus, I’m inclined to accept the paper. Though I do strongly encourage the author(s) to revise the paper as suggested above. I think making the paper easier to read will also make it more impactful as it will reach a broader audience. \n\n\n[DNS19] Duncan, A., Nuesken, N. & Szpruch, L. On the geometry of Stein variational gradient descent. arXiv.org, arXiv:1912.00894 [stat.ML], (2019).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "These are appropriately addressed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "dJ7d6k_RyFD", "reviewer": "Reviewer_ZxYc", "summary": "The authors extend gradient flow methods for variational inference to handle inequality constraints. These inequality constraints can capture fairness, safety, interpretability, and other such desiderata. They introduce two methods to solve this. First, they introduce a 'primal-dual gradient method': they formulate the equivalent minimax problem to the constrained optimization, and then they derive the dynamics for gradient descent ascent of the zero-sum game. Convergence is shown by explicitly providing a Lyapunov function. Next, they introduce a 'constraint controlled gradient descent', which first finds the feasible set, and then stays within the feasible set afterward (inspired by controlled barrier functions (CBFs) from control). Convergence is shown similarly to the work in CBFs.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This work introduces the problem of variational inference subject to inequality constraints. The tasks are new. Whereas the methods are not new, they are combined in a novel fashion to solve a new, well-motivated problem.\n\nQuality: The methods used are appropriate for the problem at hand, and the theoretical claims are (mostly) technically sound. However, one alarming issue is the fact that Assumption 3.7 is missing from the draft (it abruptly ends before the assumption is given). This MUST be fixed. However, the methods used are relatively standard and the results are intuitive; again, the main contribution of this paper is the new problem formulation.\n\nClarity: The paper is well-written; it provides a good overview of the methods, both formally and intuitively. It provides intuition for how the results are derived, as well as full rigorous proofs in the appendix. \n\nSignificance: The problem introduced is definitely of significance to the community and society at large. \n\nMajor issues: \nAssumption 3.7 is incomplete, ending with \"...we have\". This MUST be addressed.\n\nMinor notes (did not affect evaluation):\n- Typo in the title: 'Trusthworthy'\n- Line 52: 'contorl'\n- Equation 3: $\\hat y(\\theta; \\theta)$.\n- Line 86: $r_q$ should be bolded for consistency.\n- Line 108: I recommend the authors describe the median trick in greater detail.\n- Line 180: 'qudratic'\n- Line 224: 'Gaussian mixture toy' -> 'Gaussian mixture toy model'\n- Line 225: 'thrustworthy'\n- Line 285: 'manifold' -> 'manifolds'\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The work provides algorithms to include constraints in variational inference, and provides theoretical guarantees. However, as stated by the authors, the theoretical analysis primarily considers the mean-field limit. Additionally, the theorems are for continuous-time convergence, and the conditions for the discretizations to follow the continuous-time flow are not provided. Nevertheless, this paper provides a significant contribution in its own right, and these limitations are more good directions for future work rather than a shortcoming of the current paper.\n\nAdditionally, it's unclear how practical these algorithms would be in real-world settings, as the simulations provided are primarily small-scale. However, a deeper evaluation of this would be a paper in its own right, and is outside the scope of this paper, whose primary goal is theoretical development.\n", "review_text": "", "extra_scores": {"ethical_concerns": "This paper raises no ethical concerns, and seeks to address ethical concerns introduced by the deployment of machine learning methods.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that this paper should be accepted. For the camera ready: aside from generally going through the reviews/responses and ensuring that all suggested changes are made, the reviewers had one more remark for the authors. In particular, while the reviewers appreciated the effort to compare Langevin/SVGD, they still strongly suggest that the authors suitably nuance the limitations of the SVGD theory. One idea brought forward by a reviewer that the others thought was acceptable is to state results for Langevin and defer SVGD to the appendix with careful statements of limitations. This highlights that they can be obtained in a very similar manner, but that they are less likely to hold in practice.

Decision: Accept (Poster)