Novel Upper Bounds for the Constrained Most Probable Explanation Task

1 Introduction . We develop upper bounding algorithms for the constrained most probable explanation ( CMPE ) task [ 32 ] , a recently defined unifying ( discrete ) optimization task over probabilistic graphical models ( PGMs ) [ 9 , 20 , 25 ] or log-linear models . At a high level , the CMPE task adds a global capacity constraint to the classic optimization problem in PGMs called most probable explanation ( MPE ) . Given a set of log-potentials F , namely a log-linear model , the MPE task seeks to find an assignment of values to all variables ( i.e. , an explanation ) such that the sum over the projection of the assignment on the log-potentials is maximized , namely it has the maximum probability . The CMPE task adds a global constraint to MPE using a possibly different set of log-potentials G and a real number q . Specifically , it seeks to find the most probable assignment w.r.t . F under the constraint that the sum over the projection of the assignment on the log-potentials in G is smaller than or equal to q . Our interest in CMPE stems from the fact that several tasks in Explainable AI [ 16 ] can be reduced to CMPE . For example , the nearest assignment problem [ 31 ] and the decision preserving explanation problem [ 7 , 32 ] are instances of CMPE ; the former seeks to find an assignment whose probability is as close as possible to a given assignment , namely , its nearest neighbor while the latter seeks to find the most probable extension of a given partial assignment according to a generative model such that the same ( classification ) decision is made according to a discriminative model . Other applications of CMPE include various queries in robust estimation [ 10 ] and detecting adversarial attacks . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In terms of computational complexity , Rouhani et al . [ 31 , 32 ] showed that CMPE is much harder than MPE . In particular , via a reduction from the multi-choice knapsack problem ( MCKP ) [ 19 ] , Rouhani et al . showed that while MPE is linear time on graphical models having empty primal graphs , CMPE is NP-hard . However , despite these worst-case results , the good news is that MCKP is an instance of easy NP-hard problems [ 19 ] and can be well-approximated using advanced methods from the knapsack problems literature . The MCKP reduction also yields a graph-based technique for approximating CMPE . The key idea is to condition on variables , namely remove them from the primal graph G until no more than k nodes remain in each connected component . The removed nodes form a k-separator [ 3 ] and each assignment to them yields an MCKP . When the k-separator is bounded , this method yields a fully polynomial time approximation scheme ( FPTAS ) . When the k-separator is not bounded , Rouhani et al . propose to perform local search yielding an anytime algorithm . While the algorithms described above yield a lower bound , no non-trivial upper bounding algorithms are available for CMPE . Such algorithms serve two important purposes [ 24 ] : ( 1 ) generating heuristic near-optimal solutions ; and ( 2 ) pruning the search space of branch and bound methods and thus improving their efficiency . In principle , CMPE can be encoded as a mixed integer linear program ( MILP ) and solved using MILP solvers such as Gurobi [ 17 ] and SCIP [ 1 , 2 ] . However , CMPE and other related tasks such as MPE and MCKP are particularly difficult for MILP solvers because the global constraint in CMPE simultaneously restricts all the variables while MILP solvers are adept at handling sparse constraints . This motivates the development of specialized methods for CMPE . In this paper , we propose two approaches for efficiently computing qualitative upper bounds for the CMPE task . These approaches relax either the objective or the constraint or both . Our first approach is based on Lagrangian relaxation ( cf . [ 35 ] ) and relaxes the global constraint using a Lagrange multiplier λ ≥ 0 to yield an ( unconstrained ) MPE task . Given a value for λ , an upper bound on the MPE task yields an upper bound on CMPE . We propose to solve the MPE task using either exact or upper bounding approaches described in literature such as mini-bucket elimination [ 11 ] , dual decomposition [ 14 ] and join graph based cost shifting schemes [ 18 ] and further tighten the upper bound by searching for the best possible value for λ , namely a value that minimizes the upper bound on MPE . Our second approach is based on Lagrangian decomposition . The key idea is to decompose the problem by duplicating variables [ 6 ] via equality constraints and then relaxing the latter using Lagrange multipliers such that the resulting problem reduces to MCKP . Solving the MCKP for a given assignment of values to the multipliers yields an upper bound on CMPE . We propose to further improve the bound by searching for the best possible value assignment to the multipliers via an iterative algorithm . Empirically , we investigate the quality and computational efficiency of our proposed bounding techniques on a variety of CMPE tasks formulated on cutset networks [ 30 ] and graphical models used in the UAI competitions [ 13 , 15 ] . We also explore a novel application of CMPE : making classifiers ( expressed as log-linear models ) change their decision by minimally changing the test example . We found that when there is a relatively small limit on the amount of storage space an algorithm can use , our approach based on Lagrange decomposition via MCKP yields the best upper bounds . However , it also requires significantly longer to converge . Conversely , when the space limit is relatively large , the approach based on Lagrangian relaxation that utilizes MPE solvers is superior . 2 Notation and Background . We denote binary discrete random variables using upper case letters ( e.g. , X , Y , etc ) , their sets using bold uppercase letters ( e.g. , X , Y , etc . ) and assignment of values to them using bold lowercase letters ( e.g. , x , y , etc ) . If S ⊆X , then xS is the projection of x onto S. We use letters f , g and h to denote log-potentials or features and the calligraphic capital letters such as F , G , etc . to denote sets of log-potentials . A log-linear model or a Markov network , denoted byM is a pair 〈X , F〉 where X is a set of variables and F is a set of log-potentials or features such that each feature fi ∈ F is defined over a subset of variables S ( fi ) ⊆X . S ( fi ) is called the scope of fi . A log-potential maps each assignment in its scope to a real number ( R ) . A primal graph ofM is an undirected graph G ( V , E ) where V and E is the set of vertices and edges respectively such that each vertex Vi ∈ V represents the random variable Xi ∈X and there is an edge between vertices Vi and Vj in G iff the corresponding variables appear together in the scope of some function in F . Given an assignment x , the weight of x w.r.t.M , denoted by ωM ( x ) equals ∑ f∈F f ( xS ( f ) ) . For brevity , we will abuse notation and use f ( x ) instead of f ( xS ( f ) ) . Definition 1 ( CMPE ) . Given two log-linear modelsM1 = 〈X , F〉 andM2 = 〈X , G〉 defined over the same set of variables X and a real value q , the constrained most-probable explanation ( CMPE ) task is to find an assignment x∗ such that wM1 ( x ∗ ) is maximized and wM2 ( x ∗ ) ≤ q . Mathematically , the problem of computing the optimal weight is given by max x ∑ f∈F f ( x ) s.t . ∑ g∈G g ( x ) ≤ q ( 1 ) x∗ is the optimal solution to the CMPE problem . Let c∗ = ωM1 ( x ∗ ) denote the optimal value or weight . A feasible solution of CMPE is an assignment x s.t . ∑ g∈G g ( x ) ≤ q . The combined primal graph of CMPE is the graph obtained by taking the union of the edges of the primal graphs associated withM1 andM2 respectively . Formally , let G1 ( V , E1 ) and G2 ( V , E2 ) be the primal graphs associated withM1 andM2 respectively . Then the combined primal graph is G ( V , E1 ∪ E2 ) . Without loss of generality and for simplicity of exposition we assume that bothM1 andM2 have the same number of functions and for each function f ∈ F there is a corresponding function g ∈ G such that S ( f ) = S ( g ) and vice versa . Thus , under this assumption , G1 = G2 = G. Example 1 . Fig . 1 shows two log-linear models that define a CMPE problem given q = 40 . 2.1 Prior Work on Upper Bounding the Most Probable Explanation Task . Removing the constraint in Eq . ( 1 ) encoded byM2 and q yields the most probable explanation ( MPE ) task . MPE is known to be NP-hard in general but can be solved efficiently in practice when the primal graph has low treewidth or when the partition-based [ 11 ] upper bounding schemes yield close to optimal estimates within an AND/OR branch and bound framework [ 24 ] . In this paper and specifically in our experiments , we use mini-buckets based upper bounding schemes presented in Ihler et al . [ 18 ] . These schemes operate by first converting the primal graph of M to a join graph where each node called a cluster and each edge called a separator is associated and labeled with a subset of variables from M. The join graph can be thought of as a relaxation of inference architectures called tree-decompositions ( cf . [ 20 ] ) , on which MPE inference is exact , in that it satisfies all properties that the latter satisfies including the running intersection property except that it is a graph of clusters instead of a tree of clusters . Then these algorithms perform messagepassing on the edges of the join graph where the computational complexity of message-passing is exponential in the maximum cluster size . To control the complexity and yield a bounded polynomial time approximation , these schemes use an integer parameter i known as the i-bound which bounds the number of variables in each cluster in the join graph by i. Ihler et al . propose three different classes of message-passing schemes on join graphs . The first scheme which yields the classic mini-buckets ( MB ) algorithm [ 11 ] performs message-passing along a given order o of variables processing the clusters one by one . For each variable X along o , the algorithm maxes out X from each unprocessed cluster that it appears in and sends a message to all unprocessed clusters that are connected to the cluster but do not mention X . The second scheme called mini-bucket with max-marginal matching ( MM ) enhances the MB algorithm by performing LP-tightening updates on all clusters that are processed when maxing-out X . These update equations are inspired by work on a family of LP-based methods including reweighted max-product [ 34 ] , max-product linear programming ( MPLP ) [ 14 ] , dual decomposition [ 21 ] , and soft arc consistency [ 4 , 33 ] . At a high level , these tightening approaches shift cost/weight from one cluster to another without changing the original weight and thus try to re-parameterize the model in order to reduce and thus improve the upper bound . The third scheme , which we refer to as JG , is a fully iterative approach which performs message-passing or LP-tightening updates until convergence or until a bound on the number of iterations is reached . In practice , it was observed that in terms of the upper bound quality , JG is superior to MM which in turn is superior to MB . However , since MM and MB are single pass approaches while JG is iterative , JG requires significantly more time . Fig . 2 demonstrates the MB and MM algorithms using the log-linear model given in Fig . 1 having log-potentials f1 , f2 and f3 . MB is a one-pass algorithm and computes the messages denoted by h while MM also passes the marginal matching messages denoted by c ( red ) .

Summary: This paper proposes tractable strategies for upper bounding the value of constrained MPE problems. Two strategies are developed based on Lagrangian relaxation. One is to completely eliminate the constraint, converting the problem to a straight MPE problem. The second one involves the strategic application of dual decomposition to convert the problem instance to one of a multiple choice knapsack problem. The approaches are evaluated on datasets from the UAI inference challenge as well as adversarial input modification tasks.