Monotone deep Boltzmann machines

1 INTRODUCTION . This paper considers ( deep ) Boltzmann machines , which are pairwise energy-based probabilistic models . Theses models specify a joint distribution over variables x given by the density p ( x ) ∝ exp ∑ ( i , j ) ∈E x⊤i Φijxj + n∑ i=1 b⊤i xi , ( 1 ) where each x1 : n denotes a discrete random variable over ki possible values , represented as a onehot encoding xi ∈ { 0 , 1 } ki ; E denotes the set of edges in the model ; Φi , j ∈ Rki×kj represents pairwise potential ; and bi ∈ Rki represents unary potential . Depending on context , these models are typically referred to as pairwise Markov random fields ( MRFs ) ( Koller & Friedman , 2009 ) , or ( potentially deep ) Boltzmann machines ( Goodfellow et al. , 2016 ; Salakhutdinov & Hinton , 2009 ; Hinton , 2002 ) . In the above setting each xi may represent an observed or unobserved value , and there can be substantial structure within the variables ; for instance , the collection of variables x may ( and indeed will , in the main settings we consider in this paper ) consist of several different “ layers ” in a joint convolutional structure , leading to the deep convolutional Boltzmann machine ( Norouzi et al. , 2009 ) . In this paper , we propose a new parameterization and algorithmic approach to approximate inference of these probabilistic models . There are two main contributions : First , we define a generic parameterization of the pairwise kernel function Φ , that can represent a general Boltzmann machine . Our parametrization is flexible enough to incorporate almost all operators and network topology , including fully-connected layers , convolution operators , and skip-connections , etc . Through the lens of the recently developed monotone DEQ ( Winston & Kolter , 2020 ) , we constraint Φ to satisfy certain monotonicity conditions through training and inference ; this thereby assures that the mean-field approximation will always have a unique , globally-optimal fixed point under this parameterization . Second , although previous works ( Krähenbühl & Koltun , 2013 ; Baqué et al. , 2016 ) have made approaches to parallel mean-field updates , they either require strong conditions on ϕ , or fail to converge to the true mean-field distribution . We provide a properly-damped mean-field update method , based upon a generic proximal operator , which is guaranteed to converge to the mean-field fixed point , even if applied in parallel to all random variables simultaneously . Although there is no exact closed form solution of this proximal operator , we derive a very efficient Newton-based implementation . To estimate the parameters of our model , we follow the marginal-based loss minimization approach in Krähenbühl & Koltun ( 2013 ) ; Domke ( 2013 ) , where the objective is to directly maximize the likelihood induced by the approximated mean-field distribution . Such an approach grants us the ability to update parameters via taking gradient steps of the proposed mean-field iterations . With the proposed approaches , we perform both learning and inference for a deep convolutional , multi-resolution Boltzmann machine , and apply the network to model MNIST and CIFAR-10 pixels and their classes conditioned on partially observed images . Such joint probabilistic modelling allows us to simultaneously impute missing pixels and predict the class . While these are naturally a smallscale problem , we emphasize that performing joint probabilistic inference over a complete model of this type is a relatively high-dimensional task as far as traditional mean-field inference is concerned . We also compare our inference method to previous ones and demonstrate different convergence properties of each . We conclude by highlighting limitations and directions for future work with the method . 2 BACKGROUND AND RELATED WORK . This paper builds upon three main avenues of work : 1 ) deep equilibrium models , especially their convergent version , the monotone DEQ ; 2 ) the broad topic of energy-based deep model and Boltzmann machines in particular ; and 3 ) work on concave potentials and parallel methods for mean-field inference . We discuss each of these below . Equilibrium models and their provable convergence The DEQ model was first proposed by Bai et al . ( 2019 ) . Based on the observation that a neural network zt+1 = σ ( Wzt + Ux+ b ) with input injection x usually converges to a fixed point , they modeled an effectively infinite-depth network with input injection directly via its fixed point : z∗ = σ ( Wz∗ + Ux + b ) . Its backpropagation is done through the implicit function theorem and only requires constant memory . Bai et al . ( 2020 ) also showed that the multiscale DEQ models achieve near state-of-the-art performances on many large-scale tasks . Winston & Kolter ( 2020 ) later presented a parametrization of the DEQ ( denoted as monDEQ ) that guarantees provable convergence to a unique fixed point , using monotone operator theory . Specifically , they parameterize W in a way that I − W ⪰ mI ( called m-strongly monotone ) is always satisfied during training for some m > 0 ; they convert nonlinearities into proximal operators ( which include ReLU , tanh , etc . ) , and show that using existing splitting methods like forward-backward and Peaceman-Rachford can provably find the unique fixed point . Markov random field ( MRF ) and its variants MRF is a form of energy-based model , which model joint probabilities of the form pθ ( x ) = exp ( −Eθ ( x ) ) /Zθ for an energy function Eθ . A common type of MRF is the Boltzmann machine , the most successful variant of which is the restricted Boltzmann machines ( RBM ) ( Hinton , 2002 ) and its deep ( multi-layer ) variant ( Salakhutdinov & Hinton , 2009 ) . Particularly , RBMs define Eθ ( v , h ) = −a⊤v− b⊤h− v⊤Wh , where θ = { W , a , b } , v is the set of visible variables , and h is the set of latent variables . It is usually trained using the contrastive-divergence algorithm , and its inference can be done efficiently by a block mean-field approximation . However , a particular restriction of RBMs is that there can be no intra-layer connections , that is , each variable in v ( resp . h ) is independent conditioned on h ( resp . v ) . A deep RBM allows different layers of hidden nodes , but there can not be intra-layer connections . By contrast , our formulation allows intra-layer connections and is therefore is more expressive in this respect . See fig . 1 for the network topology of RBM , deep RBM , and general BM ( we also use the term general deep BM interchangeably to emphasize the existence of deep structure ) . Wu et al . ( 2016 ) proposed a deep parameterization of MRF , but their setting only considers a grid of hidden variables h and the connections among hidden units are restricted to the neighboring nodes . Therefore , it is a special case of our parameterization ( although their learning algorithm is orthogonal to ours ) . Numerous works also try to combine deep neural networks with conditional random fields ( CRF ) ( Krähenbühl & Koltun , 2013 ; Zheng et al. , 2015 ; Schwartz et al. , 2017 ) These models either train a pre-determined kernel as an RNN or use neural networks for producing either inputs or parameters of their CRFs . Parallel and convergent mean-field It is well-known that mean-field updates converge locally using a coordinate ascent algorithm ( Blei et al. , 2017 ) . However , local convergence is only guaranteed if the update is applied sequentially . Nonetheless , several works have proposed techniques to parallelize updates . Krähenbühl & Koltun ( 2013 ) proposed a concave-convex procedure ( CCCP ) to minimize the KL divergence between the true distribution and the mean-field variational family . To achieve efficient inference , they use a concave approximation to the pairwise kernel , and their fast update rule only converges if the kernel function is concave . Later , Baqué et al . ( 2016 ) derived a similar parallel damped forward iteration to ours that provably converges without the concave potential constraint . However , unlike our approach , they do not use a parameterization which ensures a global mean-field optimum , and their algorithm therefore may not converge to the actual fixed point of the mean-field updates . This is because Baqué et al . ( 2016 ) used the prox1f proximal operator ( described below ) , whereas we derive the proxαf operator to guarantee global convergence when doing mean-field updates in parallel . What ’ s more , Baqué et al . ( 2016 ) focused only on inference over prescribed potentials , and not on training the ( fully parameterized ) potentials as we do here . Lê-Huu & Alahari ( 2021 ) brought up a generalized Frank-Wolfe based framework for mean-field updates which include the methods proposed by Baqué et al . ( 2016 ) ; Krähenbühl & Koltun ( 2013 ) . Their results only guarantee global convergence to a local optimal . 3 MONOTONE DEEP BOLTZMANN MACHINES AND APPROXIMATE INFERENCE . In this section , we present the main technical contributions of this work . We begin by presenting a parameterization of the pairwise potential in a Boltzmann machine that guarantees the monotonicity condition . We then illustrate the connection between a ( joint ) mean-field inference fixed point and the fixed point of our monotone Boltzmann machine and discuss how deep structured networks can be implemented in this form practically ; this establishes that , under the monotonicity conditions on Φ , there exists a unique globally-optimal mean-field fixed point . Finally , we present an efficient parallel method for computing this mean-field fixed point , again motivated by the machinery of monotone DEQs and operator splitting methods . 3.1 A MONOTONE PARAMETERIZATION OF GENERAL BOLTZMANN MACHINES . In this section , we show how to parameterize our probabilistic model in a way that the pairwise potentials satisfy I − Φ ⪰ mI , which will be used later to show the existence of a unique meanfield fixed point . Additionally , since Φ defines a graphical model that has no self-loop , we further require Φ to be a block hollow matrix ( that is , the ki × ki diagonal blocks corresponding to each variable must be zero ) . While both these conditions on Φ are convex constraints , in practice it would be extremely difficult to project a generic set of weights onto this constraint set under an ordinary parameterization of the network . Thus , we instead advocate for a non-convex parameterization of the network weights , but one which guarantees that the monotonicity condition is always satisfied , without any constraint on the weights in the parameterization . Specifically , define the block matrix A = [ A1 A2 · · · An ] ( 2 ) with Ai ∈ Rd×ki matrices for each variables , and where d can be some arbitrarily chosen dimension . Then let Âi be a spectrally-normalized version of Ai Âi = Ai ·min { √ 1−m/∥Ai∥2 , 1 } ( 3 ) i.e. , a version of Ai normalized such that its largest singular value is at most √ 1−m ( note that we can compute the spectral norm of Ai as ∥Ai∥2 = ∥ATi Ai∥ 1/2 2 , which involves computing the singular values of only a ki × ki matrix , and thus is very fast in practice ) . We define the Â matrix analogously as the block version of these normalized matrices . Then we propose to parameterize Φ as Φ = blkdiag ( ÂT Â ) − ÂT Â ( 4 ) where blkdiag denotes the block-diagonal portion of the matrix along the ki×ki block . Put another way , this parameterizes Φ as Φij = { −ÂTi Âj if i ̸= j , 0 if i = j . ( 5 ) As the following simple theorem shows , this parameterization guarantees both hollowness of the Φ matrix and monotonicity of I −Φ , for any value of the A matrix . Theorem 3.1 . For any choice of parameters A , under the parametrization equation 4 above , we have that 1 ) Φii = 0 for all i = 1 , . . . , n , and 2 ) I −Φ ⪰ mI . Proof . Block hollowness of the matrix follows immediately from construction . To establish monotonicity , note that I −Φ ⪰ mI ⇐⇒ I + ÂT Â− blkdiag ( ÂT Â ) ⪰ mI ⇐= I − blkdiag ( ÂT Â ) ⪰ mI ⇐⇒ I − ÂTi Âi ⪰ mI , ∀i ⇐⇒ ∥Âi∥2 ≤ √ 1−m , ∀i . ( 6 ) This last property always holds by the construction of Âi .

Summary: This paper proposes a class of model called monotone deep Boltzmann machines, where the underlying potentials are parameterized (e.g., by CNNs) such that they obey some monotonicity constraint. This constraint ensures that the inference problem has a global optimum, which can be found using some generalized variant of parallel mean field. The method is inspired from monotone DEQ, previously proposed by Winston & Kolter (2020). Experiments on a joint task of image denoising and classification show that the proposed method can effectively model complex data distributions such as images.