map estimation via agreement on trees message passing and linear programming we develop and analyze methods for computing provably optimal maximum a posteriori probability map configurations for a subclass of markov random fields defined on graphs with cycles by decomposing the original distribution into a convex combination of tree structured distributions we obtain an upper bound on the optimal value of the original problem i e the log probability of the map assignment in terms of the combined optimal values of the tree problems we prove that this upper bound is tight if and only if all the tree distributions share an optimal configuration in common an important implication is that any such shared configuration must also be a map configuration for the original distribution next we develop two approaches to attempting to obtain tight upper bounds a a tree relaxed linear program lp which is derived from the lagrangian dual of the upper bounds and b a tree reweighted max product message passing algorithm that is related to but distinct from the max product algorithm in this way we establish a connection between a certain lp relaxation of the mode finding problem and a reweighted form of the max product min sum message passing algorithm

message passing for maximum weight independent set in this paper we investigate the use of message passing algorithms for the problem of finding the max weight independent set mwis in a graph first we study the performance of the classical loopy max product belief propagation we show that each fixed point estimate of max product can be mapped in a natural way to an extreme point of the linear programming lp polytope associated with the mwis problem however this extreme point may not be the one that maximizes the value of node weights the particular extreme point at final convergence depends on the initialization of max product we then show that if max product is started from the natural initialization of uninformative messages it always solves the correct lp if it converges this result is obtained via a direct analysis of the iterative algorithm and cannot be obtained by looking only at fixed points the tightness of the lp relaxation is thus necessary for max product optimality but it is not sufficient motivated by this observation we show that a simple modification of max product becomes gradient descent on a smoothed version of the dual of the lp and converges to the dual optimum we also develop a message passing algorithm that recovers the primal mwis solution from the output of the descent algorithm we show that the mwis estimate obtained using these two algorithms in conjunction is correct when the graph is bipartite and the mwis is unique finally we show that any problem of maximum a posteriori map estimation for probability distributions over finite domains can be reduced to an mwis problem we believe this reduction will yield new insights and algorithms for map estimation

information geometry on hierarchy of probability distributions an exponential family or mixture family of probability distributions has a natural hierarchical structure this paper gives an orthogonal decomposition of such a system based on information geometry a typical example is the decomposition of stochastic dependency among a number of random variables in general they have a complex structure of dependencies pairwise dependency is easily represented by correlation but it is more difficult to measure effects of pure triplewise or higher order interactions dependencies among these variables stochastic dependency is decomposed quantitatively into an orthogonal sum of pairwise triplewise and further higher order dependencies this gives a new invariant decomposition of joint entropy this problem is important for extracting intrinsic interactions in firing patterns of an ensemble of neurons and for estimating its functional connections the orthogonal decomposition is given in a wide class of hierarchical structures including both exponential and mixture families as an example we decompose the dependency in a higher order markov chain into a sum of those in various lower order markov chains

the generalized distributive law we discuss a general message passing algorithm which we call the generalized distributive law gdl the gdl is a synthesis of the work of many authors in information theory digital communications signal processing statistics and artificial intelligence it includes as special cases the baum welch algorithm the fast fourier transform fft on any finite abelian group the gallager tanner wiberg decoding algorithm viterbi s algorithm the bcjr algorithm pearl s belief propagation algorithm the shafer shenoy probability propagation algorithm and the turbo decoding algorithm although this algorithm is guaranteed to give exact answers only in certain cases the junction tree condition unfortunately not including the cases of gtw with cycles or turbo decoding there is much experimental evidence and a few theorems suggesting that it often works approximately even when it is not supposed to

rational double points on supersingular k3 surfaces we investigate configurations of rational double points with the total milnor number 21 on supersingular k3 surfaces the complete list of possible configurations is given as an application we also give the complete list of extremal quasi elliptic fibrations on supersingular k3 surfaces

scale selection for anisotropic diffusion filter by markov random field model the selection of stopping time i e scale significantly affects the performance of anisotropic diffusion filter for image denoising this paper designs a markov random field mrf scale selection model which selects scales for image segments then the denoised image is the composition of segments at their optimal scales in the scale space firstly statistics based scale selection criteria are proposed for image segments then we design a scale selection energy function in the mrf framework by considering the scale coherence between neighboring segments a segment based noise estimation algorithm is also developed to estimate the noise statistics efficiently experiments show that the performance of mrf scale selection model is much better than the previous global scale selection schemes combined with this scale selection model the anisotropic diffusion filter is comparable to or even outperform the state of the art denoising methods in performance

linear programming relaxations and belief propagation an empirical study the problem of finding the most probable map configuration in graphical models comes up in a wide range of applications in a general graphical model this problem is np hard but various approximate algorithms have been developed linear programming lp relaxations are a standard method in computer science for approximating combinatorial problems and have been used for finding the most probable assignment in small graphical models however applying this powerful method to real world problems is extremely challenging due to the large numbers of variables and constraints in the linear program tree reweighted belief propagation is a promising recent algorithm for solving lp relaxations but little is known about its running time on large problems in this paper we compare tree reweighted belief propagation trbp and powerful general purpose lp solvers cplex on relaxations of real world graphical models from the fields of computer vision and computational biology we find that trbp almost always finds the solution significantly faster than all the solvers in cplex and more importantly trbp can be applied to large scale problems for which the solvers in cplex cannot be applied using trbp we can find the map configurations in a matter of minutes for a large range of real world problems

improved moves for truncated convex models we consider the problem of obtaining an approximate maximum a posteriori estimate of a discrete random field characterized by pairwise potentials that form a truncated convex model for this problem we propose two st mincut based move making algorithms that we call range swap and range expansion our algorithms can be thought of as extensions of swap and expansion respectively that fully exploit the form of the pairwise potentials specifically instead of dealing with one or two labels at each iteration our methods explore a large search space by considering a range of labels that is an interval of consecutive labels furthermore we show that range expansion provides the same multiplicative bounds as the standard linear programming lp relaxation in polynomial time compared to previous approaches based on the lp relaxation for example interior point algorithms or tree reweighted message passing trw our methods are faster as they use only the efficient st mincut algorithm in their design we demonstrate the usefulness of the proposed approaches on both synthetic and standard real data problems

inference methods for crfs with co occurrence statistics the markov and conditional random fields crfs used in computer vision typically model only local interactions between variables as this is generally thought to be the only case that is computationally tractable in this paper we consider a class of global potentials defined over all variables in the crf we show how they can be readily optimised using standard graph cut algorithms at little extra expense compared to a standard pairwise field this result can be directly used for the problem of class based image segmentation which has seen increasing recent interest within computer vision here the aim is to assign a label to each pixel of a given image from a set of possible object classes typically these methods use random fields to model local interactions between pixels or super pixels one of the cues that helps recognition is global object co occurrence statistics a measure of which classes such as chair or motorbike are likely to occur in the same image together there have been several approaches proposed to exploit this property but all of them suffer from different limitations and typically carry a high computational cost preventing their application on large images we find that the new model we propose produces a significant improvement in the labelling compared to just using a pairwise model and that this improvement increases as the number of labels increases

adaptive methods for linear programming decoding detectability of failures of linear programming lp decoding and the potential for improvement by adding new constraints motivate the use of an adaptive approach in selecting the constraints for the underlying lp problem in this paper we make a first step in studying this method and show that by starting from a simple lp problem and adaptively adding the necessary constraints the complexity of lp decoding can be significantly reduced in particular we observe that with adaptive lp decoding the sizes of the lp problems that need to be solved become practically independent of the density of the parity check matrix we further show that adaptively adding extra constraints such as constraints based on redundant parity checks can provide large gains in the performance

structured prediction dual extragradient and bregman projections we present a simple and scalable algorithm for maximum margin estimation of structured output models including an important class of markov networks and combinatorial models we formulate the estimation problem as a convex concave saddle point problem that allows us to use simple projection methods based on the dual extragradient algorithm nesterov 2003 the projection step can be solved using dynamic programming or combinatorial algorithms for min cost convex flow depending on the structure of the problem we show that this approach provides a memory efficient alternative to formulations based on reductions to a quadratic program qp we analyze the convergence of the method and present experiments on two very different structured prediction tasks 3d image segmentation and word alignment illustrating the favorable scaling properties of our algorithm

message passing algorithms for inference and optimization belief propagation and divide and concur message passing algorithms can solve a wide variety of optimization inference and constraint satisfaction problems the algorithms operate on factor graphs that visually represent and specify the structure of the problems after describing some of their applications i survey the family of belief propagation bp algorithms beginning with a detailed description of the min sum algorithm and its exactness on tree factor graphs and then turning to a variety of more sophisticated bp algorithms including free energy based bp algorithms splitting bp algorithms that generalize tree reweighted bp and the various bp algorithms that have been proposed to deal with problems with continuous variables

signal space characterization of iterative decoding by tracing the flow of computations in the iterative decoders for low density parity check codes we formulate a signal space view for a finite number of iterations in a finite length code on a gaussian channel maximum a posteriori map codeword decoding or maximum likelihood decoding decodes to the codeword signal that is closest to the channel output in euclidean distance in contrast we show that iterative decoding decodes to the pseudosignal that has highest correlation with the channel output the set of pseudosignals corresponds to pseudocodewords only a vanishingly small number of which correspond to codewords we show that some pseudocodewords cause decoding errors but that there are also pseudocodewords that frequently correct the deleterious effects of other pseudocodewords

solution of the falkner skan equation by recursive evaluation of taylor coefficients we present a computational method for the solution of the third order boundary value problem characterized by the well known falkner skan equation on a semi infinite domain numerical treatments of this problem reported in the literature thus far are based on shooting and finite differences while maintaining the simplicity of the shooting approach the method presented in this paper uses a technique known as automatic differentiation which is neither numerical nor symbolic using automatic differentiation a taylor series solution is constructed for the initial value problems by calculating the taylor coefficients recursively the effectiveness of the method is illustrated by applying it successfully to various instances of the falkner skan equation

global minimization for continuous multiphase partitioning problems using a dual approach this paper is devoted to the optimization problem of continuous multi partitioning or multi labeling which is based on a convex relaxation of the continuous potts model in contrast to previous efforts which are tackling the optimal labeling problem in a direct manner we first propose a novel dual model and then build up a corresponding duality based approach by analyzing the dual formulation sufficient conditions are derived which show that the relaxation is often exact i e there exists optimal solutions that are also globally optimal to the original nonconvex potts model in order to deal with the nonsmooth dual problem we develop a smoothing method based on the log sum exponential function and indicate that such a smoothing approach leads to a novel smoothed primal dual model and suggests labelings with maximum entropy such a smoothing method for the dual model also yields a new thresholding scheme to obtain approximate solutions an expectation maximization like algorithm is proposed based on the smoothed formulation which is shown to be superior in efficiency compared to earlier approaches from continuous optimization numerical experiments also show that our method outperforms several competitive approaches in various aspects such as lower energies and better visual quality

numerical solution of the falkner skan equation based on quasilinearization we present two iterative methods for solving the falkner skan equation based on the quasilinearization method we formulate the original problem as a new free boundary value problem the truncated boundary depending on a small parameter is an unknown free boundary and has to be determined as part of solution using a change of variables the free boundary value problem is transformed to a problem defined on 0 1 we apply the quasilinearization method to solve the resulting nonlinear problem then we propose two different iterative algorithms by means of a cubic spline solver numerical results for various instances are compared with those reported previously in the literature the comparisons show the accuracy robustness and efficiency of the presented methodology

submodular relaxation for inference in markov random fields in this paper we address the problem of finding the most probable state of a discrete markov random field mrf also known as the mrf energy minimization problem the task is known to be np hard in general and its practical importance motivates numerous approximate algorithms we propose a submodular relaxation approach smr based on a lagrangian relaxation of the initial problem unlike the dual decomposition approach of komodakis et al 29 smr does not decompose the graph structure of the initial problem but constructs a submodular energy that is minimized within the lagrangian relaxation our approach is applicable to both pairwise and high order mrfs and allows to take into account global potentials of certain types we study theoretical properties of the proposed approach and evaluate it experimentally

roof duality complementation and persistency in quadratic 0 1 optimization the paper is concerned with the primal problem of maximizing a given quadratic pseudo boolean function four equivalent problems are discussed the primal the complementation the discrete rhys lp and the weighted stability problem of a sam graph each of them has a relaxation the roof dual the quadratic complementation the continuous rhys lp and the fractional weighted stability problem of a sam graph the main result is that the four gaps associated with the four relaxations are equal furthermore a solution to any of these problems leads at once to solutions of the other three equivalent ones the four relaxations can be solved in polynomial time by transforming them to a bipartite maximum flow problem the optimal solutions of the roof dual define best linear majorantsp x off having the following persistency property if theith coefficient inp is positive negative thenx i 1 0 in every optimum of the primal problem several characterizations are given for the case where these persistency results cannot be used to fix any variable of the primal on the other hand a class of gap free functions properly including the supermodular ones is exhibited

tree structured expectation propagation for ldpc decoding over bms channels in this paper we put forward the tree structured expectation propagation tep algorithm for decoding block and convolutional low density parity check codes over any binary channel we have already shown that tep improves belief propagation bp over the binary erasure channel bec by imposing marginal constraints over a set of pairs of variables that form a tree or a forest the tep decoder is a message passing algorithm that sequentially builds a tree forest of erased variables to capture additional information disregarded by the standard bp decoder which leads to a noticeable reduction of the error rate for finite length codes in this paper we show how the tep can be extended to any channel specifically to binary memoryless symmetric bms channels we particularly focus on how the tep algorithm can be adapted for any channel model and more importantly how to choose the tree forest to keep the gains observed for block and convolutional ldpc codes over the bec

dynamic graph cuts for efficient inference in markov random fields in this paper we present a fast new fully dynamic algorithm for the st mincut max flow problem we show how this algorithm can be used to efficiently compute map solutions for certain dynamically changing mrf models in computer vision such as image segmentation specifically given the solution of the max flow problem on a graph the dynamic algorithm efficiently computes the maximum flow in a modified version of the graph the time taken by it is roughly proportional to the total amount of change in the edge weights of the graph our experiments show that when the number of changes in the graph is small the dynamic algorithm is significantly faster than the best known static graph cut algorithm we test the performance of our algorithm on one particular problem the object background segmentation problem for video it should be noted that the application of our algorithm is not limited to the above problem the algorithm is generic and can be used to yield similar improvements in many other cases that involve dynamic change

global optimization of wavelet domain hidden markov tree for image segmentation this work presents a global energy minimization method for multiscale image segmentation using convex optimization theory the construction of energy function is motivated by the intuition that the larger the entropy the less a priori information one has on the value of the random variables first we represent the wavelet domain hidden markov tree whmt model of the original image as a structured energy function which is proved convex in marginal distributions next we derive the maximum lower bound of the energy function through lagrange dual transform for the purpose of incorporating marginal constraints into optimization finally a modified belief propagation optimization algorithm is used to perform global energy minimization of the dual convex energy function experiments on real image segmentation problems demonstrate the superior performance of this new algorithm when compared with nonconvex ones

revisiting the linear programming relaxation approach to gibbs energy minimization and weighted constraint satisfaction we present a number of contributions to the lp relaxation approach to weighted constraint satisfaction gibbs energy minimization we link this approach to many works from constraint programming which relation has so far been ignored in machine vision and learning while the approach has been mostly considered only for binary constraints we generalize it to n ary constraints in a simple and natural way this includes a simple algorithm to minimize the lp based upper bound n ary max sum diffusion however we consider using other bound optimizing algorithms as well the diffusion iteration is tractable for a certain class of high arity constraints represented as a black box which is analogical to propagators for global constraints csp diffusion exactly solves permuted n ary supermodular problems a hierarchy of gradually tighter lp relaxations is obtained simply by adding various zero constraints and coupling them in various ways to existing constraints zero constraints can be added incrementally which leads to a cutting plane algorithm the separation problem is formulated as finding an unsatisfiable subproblem of a csp

norm product belief propagation primal dual message passing for approximate inference inference problems in graphical models can be represented as a constrained optimization of a free energy function in this paper we treat both forms of probabilistic inference estimating marginal probabilities of the joint distribution and finding the most probable assignment through a unified message passing algorithm architecture in particular we generalize the belief propagation bp algorithms of sum product and max product and tree reweighted trw sum and max product algorithms trbp and introduce a new set of convergent algorithms based on convex free energy and linear programming lp relaxation as a zero temperature of a convex free energy the main idea of this work arises from taking a general perspective on the existing bp and trbp algorithms while observing that they all are reductions from the basic optimization formula of f ihi where the function f is an extended valued strictly convex but nonsmooth and the functions hi are extended valued functions not necessarily convex we use tools from convex duality to present the primal dual ascent algorithm which is an extension of the bregman successive projection scheme and is designed to handle optimization of the general type f ihi we then map the fractional free energy variational principle for approximate inference onto the optimization formula above and introduce the norm product message passing algorithm special cases of the norm product include sum product and max product bp algorithms trbp and nmplp algorithms when the fractional free energy is set to be convex convex free energy the norm product is globally convergent for the estimation of marginal probabilities and for approximating the lp relaxation we also introduce another branch of the norm product which arises as the zero temperature of the convex free energy which we refer to as the convex max product the convex max product is convergent unlike max product and aims at solving the lp relaxation

efficient semidefinite branch and cut for map mrf inference we propose a branch and cut b c method for solving general map mrf inference problems the core of our method is a very efficient bounding procedure which combines scalable semidefinite programming sdp and a cutting plane method for seeking violated constraints in order to further speed up the computation several strategies have been exploited including model reduction warm start and removal of inactive constraints we analyze the performance of the proposed method under different settings and demonstrate that our method either outperforms or performs on par with state of the art approaches especially when the connectivities are dense or when the relative magnitudes of the unary costs are low we achieve the best reported results experiments show that the proposed algorithm achieves better approximation than the state of the art methods within a variety of time budgets on challenging non submodular map mrf inference problems

fault identification via nonparametric belief propagation we consider the problem of identifying a pattern of faults from a set of noisy linear measurements unfortunately maximum a posteriori map probability estimation of the fault pattern is computationally intractable to solve the fault identification problem we propose a nonparametric belief propagation nbp approach we show empirically that our belief propagation solver is more accurate than recent state of the art algorithms including interior point methods and semidefinite programming our superior performance is explained by the fact that we take into account both the binary nature of the individual faults and the sparsity of the fault pattern arising from their rarity

guessing facets polytope structure and improved lp decoder we investigate the structure of the polytope underlying the linear programming lp decoder introduced by feldman karger and wainwright we first show that for expander codes every fractional pseudocodeword always has at least a constant fraction of nonintegral bits we then prove that for expander codes the active set of any fractional pseudocodeword is smaller by a constant fraction than that of any codeword we further exploit these geometrical properties to devise an improved decoding algorithm with the same order of complexity as lp decoding that provably performs better the method is very simple it first applies ordinary lp decoding and when it fails it proceeds by guessing facets of the polytope and then resolving the linear program on these facets while the lp decoder succeeds only if the ml codeword has the highest likelihood over all pseudocodewords we prove that the proposed algorithm when applied to suitable expander codes succeeds unless there exists a certain number of pseudocodewords all adjacent to the ml codeword on the lp decoding polytope and with higher likelihood than the ml codeword we then describe an extended algorithm still with polynomial complexity that succeeds as long as there are at most polynomially many pseudocodewords above the ml codeword

using linear programming to decode binary linear codes a new method is given for performing approximate maximum likelihood ml decoding of an arbitrary binary linear code based on observations received from any discrete memoryless symmetric channel the decoding algorithm is based on a linear programming lp relaxation that is defined by a factor graph or parity check representation of the code the resulting lp decoder generalizes our previous work on turbo like codes a precise combinatorial characterization of when the lp decoder succeeds is provided based on pseudocodewords associated with the factor graph our definition of a pseudocodeword unifies other such notions known for iterative algorithms including stopping sets irreducible closed walks trellis cycles deviation sets and graph covers the fractional distance d sub frac of a code is introduced which is a lower bound on the classical distance it is shown that the efficient lp decoder will correct up to spl lceil d sub frac 2 spl rceil 1 errors and that there are codes with d sub frac spl omega n sup 1 spl epsi an efficient algorithm to compute the fractional distance is presented experimental evidence shows a similar performance on low density parity check ldpc codes between lp decoding and the min sum and sum product algorithms methods for tightening the lp relaxation to improve performance are also provided

a spatially continuous max flow and min cut framework for binary labeling problems we propose and investigate novel max flow models in the spatially continuous setting with or without i priori defined supervised constraints under a comparative study of graph based max flow min cut we show that the continuous max flow models correspond to their respective continuous min cut models as primal and dual problems in this respect basic conceptions and terminologies from discrete max flow min cut are revisited under a new variational perspective we prove that the associated nonconvex partitioning problems unsupervised or supervised can be solved globally and exactly via the proposed convex continuous max flow and min cut models moreover we derive novel fast max flow based algorithms whose convergence can be guaranteed by standard optimization theories experiments on image segmentation both unsupervised and supervised show that our continuous max flow based algorithms outperform previous approaches in terms of efficiency and accuracy

the power of linear programming for general valued csps let d called the domain be a fixed finite set and let gamma called the valued constraint language be a fixed set of functions of the form f d m to mathbb q cup infty where different functions might have different arity m we study the valued constraint satisfaction problem parametrized by gamma denoted by vcsp gamma these are minimization problems given by n variables and the objective function given by a sum of functions from gamma each depending on a subset of the n variables for example if d 0 1 and gamma contains all ternary 0 infty valued functions vcsp gamma corresponds to 3 sat more generally if gamma contains only 0 infty valued functions vcsp gamma corresponds to csp gamma if d 0 1 and gamma contains all ternary 0 1 valued functions vcsp gamma corresponds to min 3 sat in which the goal is to minimize the number of unsatisfied clauses in a 3 cnf instance finite valued constraint languages contain functions that take on only rational values and not infinite values our main result is a precise algebraic characterization of valued constraint languages whose instances can be solved exactly by the basic linear programming relaxation blp for a valued constraint language gamma blp is a decision procedure for gamma if and only if gamma admits a symmetric fractional polymorphism of every arity for a finite valued constraint language gamma blp is a decision procedure if and only if gamma admits a symmetric fractional polymorphism of some arity or equivalently if gamma admits a symmetric fractional polymorphism of arity 2 using these results we obtain tractability of several novel classes of problems including problems over valued constraint languages that are 1 submodular on arbitrary lattices 2 k submodular on arbitrary finite domains 3 weakly and hence strongly tree submodular on arbitrary trees

ad 3 alternating directions dual decomposition for map inference in graphical models we present ad3 a new algorithm for approximate maximum a posteriori map inference on factor graphs based on the alternating directions method of multipliers like other dual decomposition algorithms ad3 has a modular architecture where local subproblems are solved independently and their solutions are gathered to compute a global update the key characteristic of ad3 is that each local subproblem has a quadratic regularizer leading to faster convergence both theoretically and in practice we provide closed form solutions for these ad3 subproblems for binary pairwise factors and factors imposing first order logic constraints for arbitrary factors large or combinatorial we introduce an active set method which requires only an oracle for computing a local map configuration making ad3 applicable to a wide range of problems experiments on synthetic and real world problems show that ad3 compares favorably with the state of the art

dynamic hybrid algorithms for map inference in discrete mrfs in this paper we present novel techniques that improve the computational and memory efficiency of algorithms for solving multi label energy functions arising from discrete mrfs or crfs these methods are motivated by the observations that the performance of minimization algorithms depends on 1 the initialization used for the primal and dual variables and 2 the number of primal variables involved in the energy function our first method dynamic expansion works by recycling results from previous problem instances the second method simplifies the energy function by reducing the number of unknown variables present in the problem further we show that it can also be used to generate a good initialization for the dynamic expansion algorithm by reusing dual variables we test the performance of our methods on energy functions encountered in the problems of stereo matching and color and object based segmentation experimental results show that our methods achieve a substantial improvement in the performance of expansion as well as other popular algorithms such as sequential tree re weighted message passing and max product belief propagation we also demonstrate the applicability of our schemes for certain higher order energy functions such as the one described for interactive texture based image and video segmentation in most cases we achieve a 10 15 times speed up in the computation time our modified expansion algorithm provides similar performance to fast pd but is conceptually much simpler both expansion and fast pd can be made orders of magnitude faster when used in conjunction with the reduce scheme proposed in this paper

high dimensional inference with random maximum a posteriori perturbations this paper presents a new approach called perturb max for high dimensional statistical inference in graphical models that is based on applying random perturbations followed by optimization this framework injects randomness into maximum a posteriori map predictors by randomly perturbing the potential function for the input a classic result from extreme value statistics asserts that perturb max operations generate unbiased samples from the gibbs distribution using high dimensional perturbations unfortunately the computational cost of generating so many high dimensional random variables can be prohibitive however when the perturbations are of low dimension sampling the perturb max prediction is as efficient as map optimization this paper shows that the expected value of perturb max inference with low dimensional perturbations can be used sequentially to generate unbiased samples from the gibbs distribution furthermore the expected value of the maximal perturbations is a natural bound on the entropy of such perturb max models a measure concentration result for perturb max values shows that the deviation of their sampled average from its expectation decays exponentially in the number of samples allowing effective approximation of the expectation

message passing algorithms for quadratic minimization gaussian belief propagation gabp is an iterative algorithm for computing the mean and variances of a multivariate gaussian distribution or equivalently the minimum of a multivariate positive definite quadratic function sufficient conditions such as walk summability that guarantee the convergence and correctness of gabp are known but gabp may fail to converge to the correct solution given an arbitrary positive definite covariance matrix as was observed by malioutov et al 2006 the gabp algorithm fails to converge if the computation trees produced by the algorithm are not positive definite in this work we will show that the failure modes of the gabp algorithm can be understood via graph covers and we prove that a parameterized generalization of the min sum algorithm can be used to ensure that the computation trees remain positive definite whenever the input matrix is positive definite we demonstrate that the resulting algorithm is closely related to other iterative schemes for quadratic minimization such as the gauss seidel and jacobi algorithms finally we observe empirically that there always exists a choice of parameters such that the above generalization of the gabp algorithm converges

maximum weight matching using odd sized cycles max product belief propagation and half integrality we study the maximum weight matching mwm problem for general graphs through the max product belief propagation bp and related linear programming lp the bp approach provides distributed heuristics for finding the maximum a posteriori map assignment in a joint probability distribution represented by a graphical model gm and respective lps can be considered as continuous relaxations of the discrete map problem it was recently shown that a bp algorithm converges to the correct map mwm assignment under a simple gm formulation of mwm as long as the corresponding lp relaxation is tight first under the motivation for forcing the tightness condition we consider a new gm formulation of mwm say c gm using non intersecting odd sized cycles in the graph the new corresponding lp relaxation say c lp becomes tight for more mwm instances however the tightness of c lp now does not guarantee such convergence and correctness of the new bp on c gm to address the issue we introduce a novel graph transformation applied to c gm which results in another gm formulation of mwm and prove that the respective bp on it converges to the correct map mwm assignment as long as c lp is tight finally we also show that c lp always has half integral solutions which leads to an efficient bp based mwm heuristic consisting of making sequential cutting plane modifications to the underlying gm our experiments show that this bp based cutting plane heuristic performs as well as that based on traditional lp solvers

bayesian helmholtz stereopsis with integrability prior helmholtz stereopsis is a 3d reconstruction method uniquely independent of surface reflectance yet its sub optimal maximum likelihood formulation with drift prone normal integration limits performance via three contributions this paper presents a complete novel pipeline for helmholtz stereopsis first we propose a bayesian formulation replacing the maximum likelihood problem by a maximum a posteriori one second a tailored prior enforcing consistency between depth and normal estimates via a novel metric related to optimal surface integrability is proposed third explicit surface integration is eliminated by taking advantage of the accuracy of prior and high resolution of the coarse to fine approach the pipeline is validated quantitatively and qualitatively against alternative formulations reaching sub millimetre accuracy and coping with complex geometry and reflectance

minimizing nonsubmodular functions with graph cuts a review optimization techniques based on graph cuts have become a standard tool for many vision applications these techniques allow to minimize efficiently certain energy functions corresponding to pairwise markov random fields mrfs currently there is an accepted view within the computer vision community that graph cuts can only be used for optimizing a limited class of mrf energies e g submodular functions in this survey we review some results that show that graph cuts can be applied to a much larger class of energy functions in particular nonsubmodular functions while these results are well known in the optimization community to our knowledge they were not used in the context of computer vision and mrf optimization we demonstrate the relevance of these results to vision on the problem of binary texture restoration

on learning conditional random fields for stereo until recently the lack of ground truth data has hindered the application of discriminative structured prediction techniques to the stereo problem in this paper we use ground truth data sets that we have recently constructed to explore different model structures and parameter learning techniques to estimate parameters in markov random fields mrfs via maximum likelihood one usually needs to perform approximate probabilistic inference conditional random fields crfs are discriminative versions of traditional mrfs we explore a number of novel crf model structures including a crf for stereo matching with an explicit occlusion model crfs require expensive inference steps for each iteration of optimization and inference is particularly slow when there are many discrete states we explore belief propagation variational message passing and graph cuts as inference methods during learning and compare with learning via pseudolikelihood to accelerate approximate inference we have developed a new method called sparse variational message passing which can reduce inference time by an order of magnitude with negligible loss in quality learning using sparse variational message passing improves upon previous approaches using graph cuts and allows efficient learning over large data sets when energy functions violate the constraints imposed by graph cuts

on the solution of falkner skan equations abstract new existence and uniqueness proofs are given for the solutions of the equations governing the self similar compressible boundary layer falkner skan equations the properties of the solutions are studied and some bounds on important quantities are concluded the paper is restricted to favourable pressure gradients and to wall cooling

hough forest random field for object recognition and segmentation this paper presents a new computational framework for detecting and segmenting object occurrences in images we combine hough forest hf and conditional random field crf into hfrf to assign labels of object classes to image regions hf captures intrinsic and contextual properties of objects crf then fuses the labeling hypotheses generated by hf for identifying every object occurrence interaction between hf and crf happens in hfrf inference which uses the metropolis hastings algorithm the metropolis hastings reversible jumps depend on two ratios of proposal and posterior distributions instead of estimating four distributions we directly compute the two ratios using hf in leaf nodes hf records class histograms of training examples and information about their configurations this evidence is used in inference for nonparametric estimation of the two distribution ratios our empirical evaluation on benchmark datasets demonstrates higher average precision rates of object detection smaller object segmentation error and faster convergence rates of our inference relative to the state of the art the paper also presents theoretical error bounds of hf and hfrf applied to a two class object detection and segmentation

numerical solution of the falkner skan equation using piecewise linear functions we present a finite element method for the solution of the falkner skan equation the method uses a coordinate transformation to map the semi infinite domain of the problem to the unit interval 0 1 by means of a suitable change of variables the transformed third order boundary value problem is further rewritten as a system of differential equations consisting of a second order equation and a first order equation the second order equation is approximated using a galerkin formulation with piecewise linear elements and the first order equation is approximated using a centered difference approximation in the process of the initial transformation a finite computational boundary is introduced this boundary is obtained as part of the computational solution by imposing an additional asymptotic boundary condition the solutions obtained thus are in excellent agreement with those obtained by previous authors

understanding the scalability of bayesian network inference using clique tree growth curves one of the main approaches to performing computation in bayesian networks bns is clique tree clustering and propagation the clique tree approach consists of propagation in a clique tree compiled from a bn and while it was introduced in the 1980s there is still a lack of understanding of how clique tree computation time depends on variations in bn size and structure in this article we improve this understanding by developing an approach to characterizing clique tree growth as a function of parameters that can be computed in polynomial time from bns specifically i the ratio of the number of a bn s non root nodes to the number of root nodes and ii the expected number of moral edges in their moral graphs analytically we partition the set of cliques in a clique tree into different sets and introduce a growth curve for the total size of each set for the special case of bipartite bns there are two sets and two growth curves a mixed clique growth curve and a root clique growth curve in experiments where random bipartite bns generated using the bpart algorithm are studied we systematically increase the out degree of the root nodes in bipartite bayesian networks by increasing the number of leaf nodes surprisingly root clique growth is well approximated by gompertz growth curves an s shaped family of curves that has previously been used to describe growth processes in biology medicine and neuroscience we believe that this research improves the understanding of the scaling behavior of clique tree clustering for a certain class of bayesian networks presents an aid for trade off studies of clique tree clustering using growth curves and ultimately provides a foundation for benchmarking and developing improved bn inference and machine learning algorithms

rounding based moves for semi metric labeling semi metric labeling is a special case of energy minimization for pairwise markov random fields the energy function consists of arbitrary unary potentials and pairwise potentials that are proportional to a given semi metric distance function over the label set popular methods for solving semi metric labeling include i move making algorithms which iteratively solve a minimum st cut problem and ii the linear programming lp relaxation based approach in order to convert the fractional solution of the lp relaxation to an integer solution several randomized rounding procedures have been developed in the literature we consider a large class of parallel rounding procedures and design move making algorithms that closely mimic them we prove that the multiplicative bound of a move making algorithm exactly matches the approximation factor of the corresponding rounding procedure for any arbitrary distance function our analysis includes all known results for move making algorithms as special cases

finding non overlapping clusters for generalized inference over graphical models graphical models use graphs to compactly capture stochastic dependencies amongst a collection of random variables inference over graphical models corresponds to finding marginal probability distributions given joint probability distributions in general this is computationally intractable which has led to a quest for finding efficient approximate inference algorithms we propose a framework for generalized inference over graphical models that can be used as a wrapper for improving the estimates of approximate inference algorithms instead of applying an inference algorithm to the original graph we apply the inference algorithm to a block graph defined as a graph in which the nodes are non overlapping clusters of nodes from the original graph this results in marginal estimates of a cluster of nodes which we further marginalize to get the marginal estimates of each node our proposed block graph construction algorithm is simple efficient and motivated by the observation that approximate inference is more accurate on graphs with longer cycles we present extensive numerical simulations that illustrate our block graph framework with a variety of inference algorithms e g those in the libdai software package these simulations show the improvements provided by our framework

combinatorial optimization of the discretized multiphase mumford shah functional the mumford shah model has been one of the most influential models in image segmentation and denoising the optimization of the multiphase mumford shah energy functional has been performed using level sets methods that optimize the mumford shah energy by evolving the level sets via the gradient descent these methods are very slow and prone to getting stuck in local optima due to the use of gradient descent after the reformulation of the 2 phase mumford shah functional on a graph several groups investigated the hierarchical extension of the graph representation to multi class the discrete hierarchical approaches are more effective than hierarchical or direct multiphase formulation using level sets however they provide approximate solutions and can diverge away from the optimal solution in this paper we present a discrete alternating optimization for the discretized vese chan approximation of the piecewise constant multiphase mumford shah functional that directly minimizes the multiphase functional without recursive bisection on the labels our approach handles the nonsubmodularity of the multiphase energy function and provides a global optimum if the image estimation data term is known apriori

on the statistical analysis of dirty pictures may 7th 1986 professor a f m smith in the chair summary a continuous two dimensional region is partitioned into a fine rectangular array of sites or pixels each pixel having a particular colour belonging to a prescribed finite set the true colouring of the region is unknown but associated with each pixel there is a possibly multivariate record which conveys imperfect information about its colour according to a known statistical model the aim is to reconstruct the true scene with the additional knowledge that pixels close together tend to have the same or similar colours in this paper it is assumed that the local characteristics of the true scene can be represented by a nondegenerate markov random field such information can be combined with the records by bayes theorem and the true scene can be estimated according to standard criteria however the computational burden is enormous and the reconstruction may reflect undesirable largescale properties of the random field thus a simple iterative method of reconstruction is proposed which does not depend on these large scale characteristics the method is illustrated by computer simulations in which the original scene is not directly related to the assumed random field some complications including parameter estimation are discussed potential applications are mentioned briefly

a study of parts based object class detection using complete graphs object detection is one of the key components in modern computer vision systems while the detection of a specific rigid object under changing viewpoints was considered hard just a few years ago current research strives to detect and recognize classes of non rigid articulated objects hampered by the omnipresent confusing information due to clutter and occlusion the focus has shifted from holistic approaches for object detection to representations of individual object parts linked by structural information along with richer contextual descriptions of object configurations along this line of research we present a practicable and expandable probabilistic framework for parts based object class representation enabling the detection of rigid and articulated object classes in arbitrary views we investigate learning of this representation from labelled training images and infer globally optimal solutions to the contextual map detection problem using a search with a novel lower bound as admissible heuristic an assessment of the inference performance of belief propagation and tree reweighted belief propagation is obtained as a by product the generality of our approach is demonstrated on four different datasets utilizing domain dependent information cues

on the optimality of solutions of the max product belief propagation algorithm in arbitrary graphs graphical models such as bayesian networks and markov random fields mrfs represent statistical dependencies of variables by a graph the max product belief propagation algorithm is a local message passing algorithm on this graph that is known to converge to a unique fixed point when the graph is a tree furthermore when the graph is a tree the assignment based on the fixed point yields the most probable values of the unobserved variables given the observed ones good empirical performance has been obtained by running the max product algorithm or the equivalent min sum algorithm on graphs with loops for applications including the decoding of turbo codes except for two simple graphs cycle codes and single loop graphs there has been little theoretical understanding of the max product algorithm on graphs with loops here we prove a result on the fixed points of max product on a graph with arbitrary topology and with arbitrary probability distributions discrete or continuous valued nodes we show that the assignment based on a fixed point is a neighborhood maximum of the posterior probability the posterior probability of the max product assignment is guaranteed to be greater than all other assignments in a particular large region around that assignment the region includes all assignments that differ from the max product assignment in any subset of nodes that form no more than a single loop in the graph in some graphs this neighborhood is exponentially large we illustrate the analysis with examples

a linear programming approach to max sum problem a review the max sum labeling problem defined as maximizing a sum of binary i e pairwise functions of discrete variables is a general np hard optimization problem with many applications such as computing the map configuration of a markov random field we review a not widely known approach to the problem developed by ukrainian researchers schlesinger et al in 1976 and show how it contributes to recent results most importantly those on the convex combination of trees and tree reweighted max product in particular we review schlesinger et al s upper bound on the max sum criterion its minimization by equivalent transformations its relation to the constraint satisfaction problem the fact that this minimization is dual to a linear programming relaxation of the original problem and the three kinds of consistency necessary for optimality of the upper bound we revisit problems with boolean variables and supermodular problems we describe two algorithms for decreasing the upper bound we present an example application for structural image analysis

on decoding irregular tanner codes with local optimality guarantees we consider decoding of binary linear tanner codes using message passing iterative decoding and linear programming lp decoding in memoryless binary input output symmetric mbios channels we present new certificates that are based on a combinatorial characterization for the local optimality of a codeword in irregular tanner codes with respect to any mbios channel this characterization is a generalization of arora proc acm symp theory of computing 2009 and vontobel proc inf theory and appl workshop 2010 and is based on a conical combination of normalized weighted subtrees in the computation trees of the tanner graph these subtrees may have any finite height h even equal or greater than half of the girth of the tanner graph in addition the degrees of local code nodes in these subtrees are not restricted to two i e these subtrees are not restricted to skinny trees we prove that local optimality in this new characterization implies maximum likelihood ml optimality and lp optimality and show that a certificate can be computed efficiently we also present a new message passing iterative decoding algorithm called normalized weighted min sum nwms nwms decoding is a belief propagation bp type algorithm that applies to any irregular binary tanner code with single parity check local codes e g low density and high density parity check codes we prove that if a locally optimal codeword with respect to height parameter h exists whereby notably h is not limited by the girth of the tanner graph then nwms decoding finds this codeword in h iterations the decoding guarantee of the nwms decoding algorithm applies whenever there exists a locally optimal codeword because local optimality of a codeword implies that it is the unique ml codeword the decoding guarantee also provides an ml certificate for this codeword finally we apply the new local optimality characterization to regular tanner codes and prove lower bounds on the noise thresholds of lp decoding in mbios channels when the noise is below these lower bounds the probability that lp decoding fails to decode the transmitted codeword decays doubly exponentially in the girth of the tanner graph

the complexity of finite valued csps we study the computational complexity of exact minimization of rational valued discrete functions let be a set of rational valued functions on a fixed finite domain such a set is called a finite valued constraint language the valued constraint satisfaction problem vcsp is the problem of minimizing a function given as a sum of functions from we establish a dichotomy theorem with respect to exact solvability for all finite valued constraint languages defined on domains of arbitrary finite size we show that every constraint language either admits a binary symmetric fractional polymorphism in which case the basic linear programming relaxation solves any instance of vcsp exactly or satisfies a simple hardness condition that allows for a polynomial time reduction from max cut to vcsp

a dual decomposition approach to feature correspondence in this paper we present a new approach for establishing correspondences between sparse image features related by an unknown nonrigid mapping and corrupted by clutter and occlusion such as points extracted from images of different instances of the same object category we formulate this matching task as an energy minimization problem by defining an elaborate objective function of the appearance and the spatial arrangement of the features optimization of this energy is an instance of graph matching which is in general an np hard problem we describe a novel graph matching optimization technique which we refer to as dual decomposition dd and demonstrate on a variety of examples that this method outperforms existing graph matching algorithms in the majority of our examples dd is able to find the global minimum within a minute the ability to globally optimize the objective allows us to accurately learn the parameters of our matching model from training examples we show on several matching tasks that our learned model yields results superior to those of state of the art methods

belief propagation and lp relaxation for weighted matching in general graphs loopy belief propagation has been employed in a wide variety of applications with great empirical success but it comes with few theoretical guarantees in this paper we analyze the performance of the max product form of belief propagation for the weighted matching problem on general graphs we show that the performance of max product is exactly characterized by the natural linear programming lp relaxation of the problem in particular we first show that if the lp relaxation has no fractional optima then max product always converges to the correct answer this establishes the extension of the recent result by bayati shah and sharma which considered bipartite graphs to general graphs perhaps more interestingly we also establish a tight converse namely that the presence of any fractional lp optimum implies that max product will fail to yield useful estimates on some of the edges we extend our results to the weighted b matching and r edge cover problems we also demonstrate how to simplify the max product message update equations for weighted matching making it easily deployable in distributed settings like wireless or sensor networks

soft arc consistency revisited the valued constraint satisfaction problem vcsp is a generic optimization problem defined by a network of local cost functions defined over discrete variables it has applications in artificial intelligence operations research bioinformatics and has been used to tackle optimization problems in other graphical models including discrete markov random fields and bayesian networks the incremental lower bounds produced by local consistency filtering are used for pruning inside branch and bound search in this paper we extend the notion of arc consistency by allowing fractional weights and by allowing several arc consistency operations to be applied simultaneously over the rationals and allowing simultaneous operations we show that an optimal arc consistency closure can theoretically be determined in polynomial time by reduction to linear programming this defines optimal soft arc consistency osac to reach a more practical algorithm we show that the existence of a sequence of arc consistency operations which increases the lower bound can be detected by establishing arc consistency in a classical constraint satisfaction problem csp derived from the original cost function network this leads to a new soft arc consistency method called virtual arc consistency which produces improved lower bounds compared with previous techniques and which can solve submodular cost functions these algorithms have been implemented and evaluated on a variety of problems including two difficult frequency assignment problems which are solved to optimality for the first time our implementation is available in the open source toulbar2 platform

an analysis of convex relaxations for map estimation of discrete mrfs the problem of obtaining the maximum a posteriori estimate of a general discrete markov random field i e a markov random field defined using a discrete set of labels is known to be np hard however due to its central importance in many applications several approximation algorithms have been proposed in the literature in this paper we present an analysis of three such algorithms based on convex relaxations i lp s the linear programming lp relaxation proposed by schlesinger 1976 for a special case and independently in chekuri et al 2001 koster et al 1998 and wainwright et al 2005 for the general case ii qp rl the quadratic programming qp relaxation of ravikumar and lafferty 2006 and iii socp ms the second order cone programming socp relaxation first proposed by muramatsu and suzuki 2003 for two label problems and later extended by kumar et al 2006 for a general label set we show that the socp ms and the qp rl relaxations are equivalent furthermore we prove that despite the flexibility in the form of the constraints objective function offered by qp and socp the lp s relaxation strictly dominates i e provides a better approximation than qp rl and socp ms we generalize these results by defining a large class of socp and equivalent qp relaxations which is dominated by the lp s relaxation based on these results we propose some novel socp relaxations which define constraints using random variables that form cycles or cliques in the graphical model representation of the random field using some examples we show that the new socp relaxations strictly dominate the previous approaches

a comparative study of energy minimization methods for markov random fields with smoothness based priors among the most exciting advances in early vision has been the development of efficient energy minimization algorithms for pixel labeling tasks such as depth or texture computation it has been known for decades that such problems can be elegantly expressed as markov random fields yet the resulting energy minimization problems have been widely viewed as intractable algorithms such as graph cuts and loopy belief propagation lbp have proven to be very powerful for example such methods form the basis for almost all the top performing stereo methods however the trade offs among different energy minimization algorithms are still not well understood in this paper we describe a set of energy minimization benchmarks and use them to compare the solution quality and runtime of several common energy minimization algorithms we investigate three promising methods graph cuts lbp and tree reweighted message passing in addition to the well known older iterated conditional mode icm algorithm our benchmark problems are drawn from published energy functions used for stereo image stitching interactive segmentation and denoising we also provide a general purpose software interface that allows vision researchers to easily switch between optimization methods the benchmarks code images and results are available at http vision middlebury edu mrf

lp decoding corrects a constant fraction of errors we show that for low density parity check ldpc codes whose tanner graphs have sufficient expansion the linear programming lp decoder of feldman karger and wainwright can correct a constant fraction of errors a random graph will have sufficient expansion with high probability and recent work shows that such graphs can be constructed efficiently a key element of our method is the use of a dual witness a zero valued dual solution to the decoding linear program whose existence proves decoding success we show that as long as no more than a certain constant fraction of the bits are flipped by the channel we can find a dual witness this new method can be used for proving bounds on the performance of any lp decoder even in a probabilistic setting our result implies that the word error rate of the lp decoder decreases exponentially in the code length under the binary symmetric channel bsc this is the first such error bound for ldpc codes using an analysis based on pseudocodewords recent work by koetter and vontobel shows that lp decoding and min sum decoding of ldpc codes are closely related by the graph cover structure of their pseudocodewords in their terminology our result implies that that there exist families of ldpc codes where the minimum bsc pseudoweight grows linearly in the block length

map inference via l2 sphere linear program reformulation maximum a posteriori map inference is an important task for graphical models due to complex dependencies among variables in realistic models finding an exact solution for map inference is often intractable thus many approximation methods have been developed among which the linear programming lp relaxation based methods show promising performance however one major drawback of lp relaxation is that it is possible to give fractional solutions instead of presenting a tighter relaxation in this work we propose a continuous but equivalent reformulation of the original map inference problem called ls lp we add the ell 2 sphere constraint onto the original lp relaxation leading to an intersected space with the local marginal polytope that is equivalent to the space of all valid integer label configurations thus ls lp is equivalent to the original map inference problem we propose a perturbed alternating direction method of multipliers admm algorithm to optimize the ls lp problem by adding a sufficiently small perturbation epsilon onto the objective function and constraints we prove that the perturbed admm algorithm globally converges to the epsilon karush kuhn tucker epsilon kkt point of the ls lp problem the convergence rate will also be analyzed experiments on several benchmark datasets from probabilistic inference challenge pic 2011 and opengm 2 show competitive performance of our proposed method against state of the art map inference methods

tree based reparameterization framework for analysis of sum product and related algorithms we present a tree based reparameterization trp framework that provides a new conceptual view of a large class of algorithms for computing approximate marginals in graphs with cycles this class includes the belief propagation bp or sum product algorithm as well as variations and extensions of bp algorithms in this class can be formulated as a sequence of reparameterization updates each of which entails refactorizing a portion of the distribution corresponding to an acyclic subgraph i e a tree or more generally a hypertree the ultimate goal is to obtain an alternative but equivalent factorization using functions that represent exact or approximate marginal distributions on cliques of the graph our framework highlights an important property of the sum product algorithm and the larger class of reparameterization algorithms the original distribution on the graph with cycles is not changed the perspective of tree based updates gives rise to a simple and intuitive characterization of the fixed points in terms of tree consistency we develop interpretations of these results in terms of information geometry the invariance of the distribution in conjunction with the fixed point characterization enables us to derive an exact expression for the difference between the true marginals on an arbitrary graph with cycles and the approximations provided by belief propagation more broadly our analysis applies to any algorithm that minimizes the bethe free energy we also develop bounds on the approximation error which illuminate the conditions that govern their accuracy finally we show how the reparameterization perspective extends naturally to generalizations of bp e g kikuchi 1951 approximations and variants via the notion of hypertree reparameterization

divide and concur and difference map bp decoders for ldpc codes the divide and concur dc algorithm introduced by gravel and elser can be considered a competitor to the belief propagation bp algorithm in that both algorithms can be applied to a wide variety of constraint satisfaction optimization and inference problems we show that dc can be interpreted as a message passing algorithm on a normal factor graph the difference map dynamics of the dc algorithm enables it to avoid traps which may be related to the trapping sets or pseudo codewords that plague bp decoders of low density parity check ldpc codes in the error floor regime we investigate two decoders for ldpc codes based on these ideas the first decoder is based directly on dc while the second decoder borrows the important difference map concept from the dc algorithm and translates it into a bp like decoder we show that this difference map belief propagation dmbp decoder has dramatically improved error floor performance compared to standard bp decoders while maintaining a similar computational complexity we present simulation results for ldpc codes comparing dc and dmbp decoders with other decoders based on sum product bp linear programming and mixed integer linear programming we also describe the close relation of the dmbp decoder to reweighted min sum algorithms including those recently proposed by ruozzi and tatikonda

message passing for graph structured linear programs proximal methods and rounding schemes the problem of computing a maximum a posteriori map configuration is a central computational challenge associated with markov random fields there has been some focus on tree based linear programming lp relaxations for the map problem this paper develops a family of super linearly convergent algorithms for solving these lps based on proximal minimization schemes using bregman divergences as with standard message passing on graphs the algorithms are distributed and exploit the underlying graphical structure and so scale well to large problems our algorithms have a double loop character with the outer loop corresponding to the proximal sequence and an inner loop of cyclic bregman projections used to compute each proximal update we establish convergence guarantees for our algorithms and illustrate their performance via some simulations we also develop two classes of rounding schemes deterministic and randomized for obtaining integral configurations from the lp solutions our deterministic rounding schemes use a re parameterization property of our algorithms so that when the lp solution is integral the map solution can be obtained even before the lp solver converges to the optimum we also propose graph structured randomized rounding schemes applicable to iterative lp solving algorithms in general we analyze the performance of and report simulations comparing these rounding schemes

discriminative learning of max sum classifiers the max sum classifier predicts n tuple of labels from n tuple of observable variables by maximizing a sum of quality functions defined over neighbouring pairs of labels and observable variables predicting labels as map assignments of a random markov field is a particular example of the max sum classifier learning parameters of the max sum classifier is a challenging problem because even computing the response of such classifier is np complete in general estimating parameters using the maximum likelihood approach is feasible only for a subclass of max sum classifiers with an acyclic structure of neighbouring pairs recently the discriminative methods represented by the perceptron and the support vector machines originally designed for binary linear classifiers have been extended for learning some subclasses of the max sum classifier besides the max sum classifiers with the acyclic neighbouring structure it has been shown that the discriminative learning is possible even with arbitrary neighbouring structure provided the quality functions fulfill some additional constraints in this article we extend the discriminative approach to other three classes of max sum classifiers with an arbitrary neighbourhood structure we derive learning algorithms for two subclasses of max sum classifiers whose response can be computed in polynomial time i the max sum classifiers with supermodular quality functions and ii the max sum classifiers whose response can be computed exactly by a linear programming relaxation moreover we show that the learning problem can be approximately solved even for a general max sum classifier

multiscale stochastic modeling for tractable inference and data assimilation we consider a class of multiscale gaussian models on pyramidally structured graphs while such models have been considered in the past very recent advances in inference methods for graphical models not only yield additional motivation for this class of models but also bring techniques that lead to new and powerful algorithms we provide a brief summary of these recent advances including so called walk sum analysis methods based on lagrangian relaxation and a new method for low rank wavelet based unbiased estimation of error variances and then adapt and apply them to problems of estimation for pyramidal models we demonstrate that our models not only capture long range dependencies but that they also have the property that conditioned on neighboring scales the correlation behavior within a scale is dramatically compressed this leads to algorithms resembling multipole methods for solving partial differential equations in which we alternate computations across scale using an embedded tree in the pyramidal graph with local updates within each scale not only are these algorithms guaranteed to converge to the correct answers but they also lead to new adaptive methods for choosing embedded trees and subgraphs to achieve rapid convergence this approach also leads to a solution to the so called re estimation problem in which we seek to update an estimate rapidly after local changes are made to the prior model or to the available data in addition by using a consistent probabilistic model across as well as within scales we are able both to exploit low rank variance estimation methods and to develop efficient iterative algorithms for parameter estimation

convergence of weighted min sum decoding via dynamic programming on trees applying the max product and sum product algorithms to loopy graphs is now quite popular for best assignment problems this is largely due to their low computational complexity and impressive performance in practice still there is no general understanding of the conditions required for convergence or optimality of converged solutions or both this paper presents an analysis of both attenuated max product decoding and weighted min sum decoding for low density parity check ldpc codes which guarantees convergence to a fixed point when a weight factor is sufficiently small it also shows that if the fixed point satisfies some consistency conditions then it must be both a linear programming lp and maximum likelihood ml decoding solution for dv dc regular ldpc codes the weight factor must satisfy dv 1 1 are given finally connections are explored with recent work on the threshold of lp decoding

a survey and comparison of discrete and continuous multi label optimization approaches for the potts model we present a survey and a comparison of a variety of algorithms that have been proposed over the years to minimize multi label optimization problems based on the potts model discrete approaches based on markov random fields as well as continuous optimization approaches based on partial differential equations can be applied to the task in contrast to the case of binary labeling the multi label problem is known to be np hard and thus one can only expect near optimal solutions in this paper we carry out a theoretical comparison and an experimental analysis of existing approaches with respect to accuracy optimality and runtime aimed at bringing out the advantages and short comings of the respective algorithms systematic quantitative comparison is done on the graz interactive image segmentation benchmark this paper thereby generalizes a previous experimental comparison klodt et al 2008 from the binary to the multi label case

large scale binary quadratic optimization using semidefinite relaxation and applications in computer vision many problems can be formulated as binary quadratic programs bqps which are in general np hard finding a solution when the problem is of large size to be of practical interest typically requires relaxation semidefinite relaxation usually yields tight bounds but its computational complexity is high in this work we present a semidefinite programming sdp formulation for bqps with two desirable properties first it produces similar bounds to the standard sdp formulation second compared with the conventional sdp formulation the proposed sdp formulation leads to a considerably more efficient and scalable dual optimization approach we then propose two solvers namely quasi newton and smoothing newton methods for the simplified dual problem both of them are significantly more efficient than standard interior point methods empirically the smoothing newton solver is faster than the quasi newton solver for dense or medium sized problems while the quasi newton solver is preferable for large sparse structured problems

a novel approach to the numerical solution of boundary value problems on infinite intervals the classical numerical treatment of two point boundary value problems on infinite intervals is based on the introduction of a truncated boundary instead of infinity where appropriate boundary conditions are imposed then the truncated boundary allowing for a satisfactory accuracy is computed by trial motivated by several problems of interest in boundary layer theory here we consider boundary value problems on infinite intervals governed by a third order ordinary differential equation we highlight a novel approach to define the truncated boundary the main result is the convergence of the solution of our formulation to the solution of the original problem as a suitable parameter goes to zero in the proposed formulation the truncated boundary is an unknown free boundary and has to be determined as part of the solution for the numerical solution of the free boundary formulation a noniterative and an iterative transformation method are introduced furthermore we characterize the class of free boundary value problems that can be solved noniteratively a nonlinear flow problem involving two physical parameters and belonging to the characterized class of problems is then solved moreover the falkner skan equation with relevant boundary conditions is considered and representative results obtained by the iterative transformation method are listed for the homann flow all the obtained numerical results clearly indicate the effectiveness of our approach finally we discuss the possible extensions of the proposed approach and for the question of a priori error analysis

joint sulcal detection on cortical surfaces with graphical models and boosted priors in this paper we propose an automated approach for the joint detection of major sulci on cortical surfaces by representing sulci as nodes in a graphical model we incorporate markovian relations between sulci and formulate their detection as a maximum a posteriori map estimation problem over the joint space of major sulci to make the inference tractable a sample space with a finite number of candidate curves is automatically generated at each node based on the hamilton jacobi skeleton of sulcal regions using the adaboost algorithm we learn both individual and pairwise shape priors of sulcal curves from training data which are then used to define potential functions in the graphical model based on the connection between adaboost and logistic regression finally belief propagation is used to perform the map inference and select the joint detection results from the sample spaces of candidate curves in our experiments we quantitatively validate our algorithm with manually traced curves and demonstrate the automatically detected curves can capture the main body of sulci very accurately a comparison with independently detected results is also conducted to illustrate the advantage of the joint detection approach

pencils of curves on smooth surfaces although the theory of singularities of curves resolution classification numerical invariants goes through with comparatively little change in finite characteristic pencils of curves are more difficult bertini s theorem only holds in a much weaker form and it is convenient to restrict to pencils such that when all base points are resolved the general member of the pencil becomes non singular even here the usual rule for calculating the euler characteristic of the resolved surface has to be modified by a term measuring wild ramification we begin by describing this background then proceed to discuss the exceptional members of a pencil in characteristic 0 it was shown by ha and le and by le and weber using topological reasoning that exceptional members can be characterised by their euler characteristics we present a combinatorial argument giving a corresponding result in characteristic p we first treat pencils with no base points and then reduce the remaining case to this

a new class of upper bounds on the log partition function we introduce a new class of upper bounds on the log partition function of a markov random field mrf this quantity plays an important role in various contexts including approximating marginal distributions parameter estimation combinatorial enumeration statistical decision theory and large deviations bounds our derivation is based on concepts from convex duality and information geometry in particular it exploits mixtures of distributions in the exponential domain and the legendre mapping between exponential and mean parameters in the special case of convex combinations of tree structured distributions we obtain a family of variational problems similar to the bethe variational problem but distinguished by the following desirable properties i they are convex and have a unique global optimum and ii the optimum gives an upper bound on the log partition function this optimum is defined by stationary conditions very similar to those defining fixed points of the sum product algorithm or more generally any local optimum of the bethe variational problem as with sum product fixed points the elements of the optimizing argument can be used as approximations to the marginals of the original model the analysis extends naturally to convex combinations of hypertree structured distributions thereby establishing links to kikuchi approximations and variants

partial optimality by pruning for map inference with general graphical models we consider the energy minimization problem for undirected graphical models also known as map inference problem for markov random fields which is np hard in general we propose a novel polynomial time algorithm to obtain a part of its optimal non relaxed integral solution our algorithm is initialized with variables taking integral values in the solution of a convex relaxation of the map inference problem and iteratively prunes those which do not satisfy our criterion for partial optimality we show that our pruning strategy is in a certain sense theoretically optimal also empirically our method outperforms previous approaches in terms of the number of persistently labelled variables the method is very general as it is applicable to models with arbitrary factors of an arbitrary order and can employ any solver for the considered relaxed problem our method s runtime is determined by the runtime of the convex relaxation solver for the map inference problem

hybrid linear programming based decoding algorithm for ldpc codes this paper presents a hybrid decoding algorithm for low density parity check ldpc codes based on the interior point method with barrier function for linear programming lp decoding introduced by wadayama first an efficient implementation of wadayama s algorithm is presented the main idea behind the modification is to approximate the barrier function for the fundamental polytope defining the code so that it contains only one linear constraint for each of the parity check constraints a two stage hybrid decoding which combines the interior point decoding and a low complexity decoding algorithm for ldpc codes is then proposed simulation results show that the approximations introduced in the proposed algorithms do not result in any performance degradation while considerably reducing the decoding complexity and latency