Regularized Frank-Wolfe for Dense CRFs: Generalizing Mean Field and Beyond

1 Introduction . Fully-connected or dense conditional random fields ( CRFs ) [ 34 ] —combined with strong pixel-level classifiers such as a convolutional neural network ( CNN ) [ 25 , 42 ] —have been a highly-successful paradigm for semantic segmentation . Top-performing systems on the PASCAL VOC benchmark [ 22 ] used to include a CRF as either a post-processing step [ 13 , 14 , 15 , 20 , 43 , 44 , 45 ] or a trainable component [ 4 , 49 , 64 , 68 , 73 , 76 ] . However , as CNNs got stronger , the improvements that CRFs brought decreased over time , and as a result they fell out of favor since 2017 [ 45 ] . In this paper , we revisit dense CRFs with two contributions . First , on the theoretical side , we propose regularized Frank-Wolfe , a new class of algorithms for inference and learning of CRFs that perform better than the popular mean field [ 34 , 35 , 58 ] —the method of choice in the aforementioned works . Regularized Frank-Wolfe optimizes a nonconvex continuous relaxation of the CRF inference problem ( §2 ) by performing approximate conditional-gradient updates ( §3.1 ) , which is equivalent to minimizing a regularized energy using the generalized Frank-Wolfe method [ 53 ] ( §3.2 ) . Several of its instantiations lead to new algorithms that have not been studied before in the MAP inference literature ( §3.3 ) . Moreover , we show that it also includes several existing methods , including mean field and the concave-convex procedure [ 75 ] , as special cases ( §3.4 ) . This generalized perspective allows a unified analysis of all these old and new algorithms in a single framework ( §4 ) . In particular , we show that they achieve a sublinear rate of convergence O ( 1/ p k ) for suitable stepsize schemes , and in certain cases ( such as strongly-convex regularizer or concave energy ) this can be improved to O ( 1/k ) ( §4.1 ) . Furthermore , we provide a tightness analysis for the resulting nonconvex relaxation of the regularized energy , recovering some existing tightness results [ 10 , 41 , 61 ] as special cases ( §4.2 ) . The proposed algorithms are easy to implement , converge quickly in practice , and have ( sub ) differentiable iterates . Such properties are important for successful gradient-based learning via backpropagation [ 47 , 62 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Our second contribution lies on the practical side . In addition to mean field and regularized FrankWolfe variants , we re-implement several existing first-order inference methods [ 39 , 41 , 48 ] —those that are amenable to gradient-based learning—for comparison . Remarkably , we find that dense CRFs can still achieve important improvements over the strong DeepLabv3+ [ 17 ] CNN model for all these solvers ( §5 ) . In particular , our best variant of regularized Frank-Wolfe achieves a mean intersection-over-union ( mIoU ) score of 88.0 on the PASCAL VOC test set ( §5.3 ) , improving over DeepLabv3+ . We hope that these encouraging results could attract interest from the community in considering dense CRFs ( again ) for tasks such as semantic segmentation . Our source code is made publicly available under the GNU general public license for this purpose.1 2 Background . 2.1 Inference in CRFs . Let s 2 S1 ⇥ · · · ⇥ Sn denote an assignment to n discrete random variables S1 , . . . , Sn , where each variable Si takes values in a finite set of states ( or labels ) Si . Let G = ( V , E ) be a graph of n nodes ( V = { 1 , 2 , . . . , n } ) . A Markov random field ( MRF ) defined by G encodes a family of joint distributions that can be factorized as follows , where i : Si ! R+ and ij : Si ⇥ Sj ! R+ are the so-called unary and pairwise ( respectively ) potential functions , and Z is a normalization factor : p ( s ) = 1 Z Y i2V i ( si ) Y ij2E ij ( si , sj ) . ( 1 ) Note that ( 1 ) can also include conditional distributions , i.e. , p ( s |o ) with observed variables o . In this case the potentials may also depend on o , e.g. , i ( si ; o ) , and this model is referred to as conditional random field ( CRF ) [ 37 ] . We will present later ( §5.1 ) such a model for image segmentation . In the following , we use MRF and CRF interchangeably . We assume further that all nodes have the same set of labels : Si = S 8i , with cardinality d = |S| . It is convenient to express p ( s ) as 1Z exp ( e ( s ) ) , where the so-called energy e ( s ) is defined as e ( s ) = X i2V ✓i ( si ) + X ij2E ✓ij ( si , sj ) , with ✓i ( si ) = log i ( si ) ( idem for ✓ij ) . ( 2 ) The task of maximum a posteriori ( MAP ) inference consists in finding the most probable joint assignment , also known as energy minimization ( which is NP-Hard in general [ 65 ] ) : s⇤ = argmax s2Sn p ( s ) = argmin s2Sn e ( s ) . ( 3 ) 2.2 Continuous relaxation of MAP inference . Let xis be a binary variable such that xis = 1 iff label s is assigned to node i . Then , xi = ( xis ) s2S 2 { 0 , 1 } d denotes the one-hot vector for node i . Let x 2 { 0 , 1 } nd be the concatenation of all xi , and let ✓i = ( ✓i ( s ) ) s2S 2 Rd , ⇥ij = ( ✓ij ( s , t ) ) s2St2S 2 Rd⇥d . The energy ( 2 ) is then E ( x ; ✓ ) = X i2V ✓ > i xi + X ij2E x > i ⇥ijxj , ( 4 ) where ✓ is a parameter vector composed of all ✓i and⇥ij . The MAP inference problem ( 3 ) transforms to minimizing E ( x ; ✓ ) over the new variables x . A natural approach is to relax the binary constraint and solve the continuous relaxation minx2X E ( x ; ✓ ) with X = x 2 Rnd : x 0,1 > xi = 1 8i 2 V . ( 5 ) This continuous relaxation is known to be tight [ 10 , 41 , 61 ] . For convenience , we represent the unary potentials as a vector u ( ✓ ) = ( ✓i ) i2V 2 Rnd , and the pairwise potentials as a symmetric n⇥ n block matrix P ( ✓ ) 2 Rnd⇥nd , where the ( i , j ) block is ⇥ij . The problem is reduced to min x2X E ( x ; ✓ ) , 1 2 x > P ( ✓ ) x+ u ( ✓ ) > x . ( 6 ) 1https : //github.com/netw0rkf10w/CRF The reason we have made ✓ explicit in ( 4 ) and ( 6 ) is to provide more clarity when discussing the differentiability of their solutions for the learning task . Note that an optimal solution x⇤ to ( 4 ) and ( 6 ) is a function of ✓ , and thus should be written as x⇤ ( ✓ ) . Therefore , when we say a solution is differentiable , it is understood that it is so with respect to ✓ . When there is no ambiguity , we omit ✓ and write simply E ( x ) , P , and u . We will be interested in problem ( 6 ) in the rest of the paper , though it should be noted that our method also applies to the so-called linear programming ( LP ) relaxation , which takes the form minx2XLP ELP ( x ; ✓ ) , where ELP ( x ; ✓ ) = ✓ > x and XLP is the so-called local polytope [ 72 ] . We refer to Appendix A for the details . 2.3 Vanilla Frank-Wolfe algorithm for MAP inference . Since the continuous energy is differentiable , it is natural to apply first-order methods such as FrankWolfe [ 23 ] to solving ( 6 ) [ 41 ] . Starting from a feasible x0 2 X , Frank-Wolfe approximately solves ( 6 ) by iterating the following steps , where ↵k 2 [ 0 , 1 ] follows some stepsize scheme : pk 2 argmin p2X ⌦ rE ( xk ) , p ↵ , xk+1 = xk + ↵k ( p k xk ) . ( 7 ) The same idea has also been successfully applied to other types of continuous relaxations of ( 6 ) , such as linear programming ( LP ) [ 52 ] or convex quadratic programming ( QP ) [ 21 , 61 ] . 3 Regularized Frank-Wolfe for inference . In this section , we introduce the proposed regularized Frank-Wolfe as a general class of algorithms for MAP inference . The motivation and general framework are presented in §3.1 and §3.2 . We show that several new inference algorithms can be obtained using different regularizers ( §3.3 ) . Moreover , regularized Frank-Wolfe also includes a number of existing algorithms as special cases ( §3.4 ) . 3.1 A smoothing perspective . We have seen in §2.3 the ( vanilla ) Frank-Wolfe method for solving MAP inference . Unfortunately , from a learning perspective , this algorithm is problematic . Indeed , CRF learning with stochastic gradient descent ( SGD ) is typically done by backpropagating through the optimization steps [ 35 , 63 , 76 ] , but the iterate pk in ( 7 ) is piecewise constant and thus its gradient ( w.r.t . ✓ ) is zero almost everywhere ( §C.1 ) , which makes learning not possible.2 This issue will be illustrated later in the experiments . A potential solution is to add a ( typically strongly- ) convex regularization term : pk 2 argmin p2X ⌦ rE ( xk ) , p ↵ + r ( p ) , xk+1 = xk + ↵k ( p k xk ) . ( 8 ) With appropriately chosen regularizers , ( 8 ) becomes suitable for gradient-based learning . The technique of approximating an update step by a regularized one is quite standard in the optimization literature . For example , the classical proximal gradient method [ 48 ] can be interpreted the same way . Our update ( 8 ) is inspired by Nesterov ’ s smoothing [ 56 ] , and thus is similar in spirit to its many applications [ 31 , 57 , 66 ] . In the next section , we present a different , more general perspective to view ( 8 ) , which offers more flexibility in designing new algorithms , in making connections to existing ones , and in analyzing their theoretical properties in a unified manner . 3.2 A regularized energy perspective . Instead of approximating the local updates of a given algorithm ( in this case : vanilla Frank-Wolfe ) to minimize the same objective function , one may choose to keep the algorithm , and approximate instead the objective . At first glance , however , this idea does not seem to be a good one . Indeed , if we replace E by some function E0 and apply vanilla Frank-Wolfe , the updates ( 7 ) remains piecewise constant , thus we have the same zero-gradient issue . It turns out that , if we choose a more appropriate “ given algorithm ” , this idea can work . Such a choice is the generalized Frank-Wolfe algorithm [ 5 , 12 , 53 ] . 2Note that backpropagation typically requires the operations to be differentiable ( at least ) almost everywhere , which pk satisfies . Therefore , the issue here does not lie in differentiability , but in the resulting zero gradients . Blackbox differentiation [ 60 ] can deal with this scenario , but it is limited to LP relaxations . Consider the following problem , where f : Rm ! R [ { +1 } is differentiable but possibly nonconvex , and g : Rm ! R [ { +1 } is proper , closed , and convex but possibly non-differentiable : min x F ( x ) , f ( x ) + g ( x ) . ( 9 ) Generalized Frank-Wolfe solves ( 9 ) by iterating pk 2 argmin p ⌦ rf ( xk ) , p ↵ + g ( p ) , xk+1 = xk + ↵k ( p k xk ) . ( 10 ) If g is the indicator function X of X ( i.e. , X ( x ) = 0 if x 2 X and X ( x ) = +1 otherwise ) , then the algorithm clearly reduces to vanilla Frank-Wolfe ( 7 ) for minx2X f ( x ) . Therefore , generalized Frank-Wolfe applied to ( 6 ) with f = E and g = X will yield exactly the same updates ( 7 ) . Now let us apply this algorithm to an approximate objective Er ( x ) = E ( x ) + r ( x ) for some function r. Choosing f = E and g = r + X , it is straightforward that ( 10 ) reduces to ( 8 ) . Therefore , we have recovered the same algorithm as in §3.1 , but this time through different machinery . This framework offers a great flexibility as one can choose f and g in many different ways to obtain new algorithms . The only conditions are f being differentiable and g being convex , so that the subproblem in ( 10 ) is well-defined and globally solvable.3 For example , instead of choosing f = E and g = r+ X as above , one can choose f ( x ) = 12x > Px and g ( x ) = u > x+r ( x ) + X ( x ) . We will recover later in §3.4 some existing algorithms ( as special cases ) through this kind of decomposition . Finally , we present Algorithm 1 for ( approximately ) solving MAP inference ( 6 ) . Algorithm 1 Generic regularized Frank-Wolfe for ( approximately ) solving MAP inference ( 6 ) . 1 : Choose a regularizer r such that there exist f ( differentiable ) and g ( convex ) satisfying f + g = E + r + X . Typically ( but not necessarily ) r is convex on X and is constant on X \ { 0 , 1 } nd . 2 : Initialization : k 0 , x0 2 X , number of iterations N . 3 : Compute pk 2 argminp ⌦ rf ( xk ) , p ↵ + g ( p ) and compute the stepsize ↵k . 4 : Update xk+1 = xk + ↵k ( pk xk ) . Let k k + 1 and go to Step 3 until k = N . 5 : Rounding : convert x to a discrete solution and return . While the choice of ( r , f , g ) can be highly flexible , it would make little sense to optimize a function that has nothing to do with the original objective ( i.e. , the discrete energy ) . Let X = X \ { 0 , 1 } nd denote the discrete domain of our problem . If we choose r such that it is constant on X ( as suggested in Step 1 above ) , then minimizing E on X is equivalent to minimizing E+r on X , and thus Algorithm 1 actually solves the continuous relaxation of a ( different ) discrete problem that is equivalent to MAP inference . Further discussion on this matter , as well as on the rounding Step 5 , are deferred until §4.2 . Finally , we should note that adding a strongly-convex regularizer is not new in the MAP inference literature [ 29 , 31 , 52 , 67 , 69 ] . In particular , some previous work even applied ( vanilla ) Frank-Wolfe to optimizing such regularized energy [ 52 , 67 , 69 ] . All these algorithms , however , suffer from the aforementioned zero-gradient issue , as already explained in the beginning of this section .

Summary: Semantic segmentation is a widely-studied problem in computer vision, and over the years a range of techniques have been popular. This paper re-visits the use of dense CRF layers on top of a neural network, which can be used to model pairwise connectivity across output labels. They contribute a new optimization method for mean field inference in the CRF that converges quickly (provably), and is differentiable (such that the whole model can be trained end-to-end). The paper pushes the SOTA on two popular segmentation benchmarks.