Monotone deep Boltzmann machines

1 INTRODUCTION . This paper considers ( deep ) Boltzmann machines , which are pairwise energy-based probabilistic models . Theses models specify a joint distribution over variables x given by the density p ( x ) ∝ exp ∑ ( i , j ) ∈E x⊤i Φijxj + n∑ i=1 b⊤i xi , ( 1 ) where each x1 : n denotes a discrete random variable over ki possible values , represented as a onehot encoding xi ∈ { 0 , 1 } ki ; E denotes the set of edges in the model ; Φi , j ∈ Rki×kj represents pairwise potential ; and bi ∈ Rki represents unary potential . Depending on context , these models are typically referred to as pairwise Markov random fields ( MRFs ) ( Koller & Friedman , 2009 ) , or ( potentially deep ) Boltzmann machines ( Goodfellow et al. , 2016 ; Salakhutdinov & Hinton , 2009 ; Hinton , 2002 ) . In the above setting each xi may represent an observed or unobserved value , and there can be substantial structure within the variables ; for instance , the collection of variables x may ( and indeed will , in the main settings we consider in this paper ) consist of several different “ layers ” in a joint convolutional structure , leading to the deep convolutional Boltzmann machine ( Norouzi et al. , 2009 ) . In this paper , we propose a new parameterization and algorithmic approach to approximate inference of these probabilistic models . There are two main contributions : First , we define a generic parameterization of the pairwise kernel function Φ , that can represent a general Boltzmann machine . Our parametrization is flexible enough to incorporate almost all operators and network topology , including fully-connected layers , convolution operators , and skip-connections , etc . Through the lens of the recently developed monotone DEQ ( Winston & Kolter , 2020 ) , we constraint Φ to satisfy certain monotonicity conditions through training and inference ; this thereby assures that the mean-field approximation will always have a unique , globally-optimal fixed point under this parameterization . Second , although previous works ( Krähenbühl & Koltun , 2013 ; Baqué et al. , 2016 ) have made approaches to parallel mean-field updates , they either require strong conditions on ϕ , or fail to converge to the true mean-field distribution . We provide a properly-damped mean-field update method , based upon a generic proximal operator , which is guaranteed to converge to the mean-field fixed point , even if applied in parallel to all random variables simultaneously . Although there is no exact closed form solution of this proximal operator , we derive a very efficient Newton-based implementation . To estimate the parameters of our model , we follow the marginal-based loss minimization approach in Krähenbühl & Koltun ( 2013 ) ; Domke ( 2013 ) , where the objective is to directly maximize the likelihood induced by the approximated mean-field distribution . Such an approach grants us the ability to update parameters via taking gradient steps of the proposed mean-field iterations . With the proposed approaches , we perform both learning and inference for a deep convolutional , multi-resolution Boltzmann machine , and apply the network to model MNIST and CIFAR-10 pixels and their classes conditioned on partially observed images . Such joint probabilistic modelling allows us to simultaneously impute missing pixels and predict the class . While these are naturally a smallscale problem , we emphasize that performing joint probabilistic inference over a complete model of this type is a relatively high-dimensional task as far as traditional mean-field inference is concerned . We also compare our inference method to previous ones and demonstrate different convergence properties of each . We conclude by highlighting limitations and directions for future work with the method . 2 BACKGROUND AND RELATED WORK . This paper builds upon three main avenues of work : 1 ) deep equilibrium models , especially their convergent version , the monotone DEQ ; 2 ) the broad topic of energy-based deep model and Boltzmann machines in particular ; and 3 ) work on concave potentials and parallel methods for mean-field inference . We discuss each of these below . Equilibrium models and their provable convergence The DEQ model was first proposed by Bai et al . ( 2019 ) . Based on the observation that a neural network zt+1 = σ ( Wzt + Ux+ b ) with input injection x usually converges to a fixed point , they modeled an effectively infinite-depth network with input injection directly via its fixed point : z∗ = σ ( Wz∗ + Ux + b ) . Its backpropagation is done through the implicit function theorem and only requires constant memory . Bai et al . ( 2020 ) also showed that the multiscale DEQ models achieve near state-of-the-art performances on many large-scale tasks . Winston & Kolter ( 2020 ) later presented a parametrization of the DEQ ( denoted as monDEQ ) that guarantees provable convergence to a unique fixed point , using monotone operator theory . Specifically , they parameterize W in a way that I − W ⪰ mI ( called m-strongly monotone ) is always satisfied during training for some m > 0 ; they convert nonlinearities into proximal operators ( which include ReLU , tanh , etc . ) , and show that using existing splitting methods like forward-backward and Peaceman-Rachford can provably find the unique fixed point . Markov random field ( MRF ) and its variants MRF is a form of energy-based model , which model joint probabilities of the form pθ ( x ) = exp ( −Eθ ( x ) ) /Zθ for an energy function Eθ . A common type of MRF is the Boltzmann machine , the most successful variant of which is the restricted Boltzmann machines ( RBM ) ( Hinton , 2002 ) and its deep ( multi-layer ) variant ( Salakhutdinov & Hinton , 2009 ) . Particularly , RBMs define Eθ ( v , h ) = −a⊤v− b⊤h− v⊤Wh , where θ = { W , a , b } , v is the set of visible variables , and h is the set of latent variables . It is usually trained using the contrastive-divergence algorithm , and its inference can be done efficiently by a block mean-field approximation . However , a particular restriction of RBMs is that there can be no intra-layer connections , that is , each variable in v ( resp . h ) is independent conditioned on h ( resp . v ) . A deep RBM allows different layers of hidden nodes , but there can not be intra-layer connections . By contrast , our formulation allows intra-layer connections and is therefore is more expressive in this respect . See fig . 1 for the network topology of RBM , deep RBM , and general BM ( we also use the term general deep BM interchangeably to emphasize the existence of deep structure ) . Wu et al . ( 2016 ) proposed a deep parameterization of MRF , but their setting only considers a grid of hidden variables h and the connections among hidden units are restricted to the neighboring nodes . Therefore , it is a special case of our parameterization ( although their learning algorithm is orthogonal to ours ) . Numerous works also try to combine deep neural networks with conditional random fields ( CRF ) ( Krähenbühl & Koltun , 2013 ; Zheng et al. , 2015 ; Schwartz et al. , 2017 ) These models either train a pre-determined kernel as an RNN or use neural networks for producing either inputs or parameters of their CRFs . Parallel and convergent mean-field It is well-known that mean-field updates converge locally using a coordinate ascent algorithm ( Blei et al. , 2017 ) . However , local convergence is only guaranteed if the update is applied sequentially . Nonetheless , several works have proposed techniques to parallelize updates . Krähenbühl & Koltun ( 2013 ) proposed a concave-convex procedure ( CCCP ) to minimize the KL divergence between the true distribution and the mean-field variational family . To achieve efficient inference , they use a concave approximation to the pairwise kernel , and their fast update rule only converges if the kernel function is concave . Later , Baqué et al . ( 2016 ) derived a similar parallel damped forward iteration to ours that provably converges without the concave potential constraint . However , unlike our approach , they do not use a parameterization which ensures a global mean-field optimum , and their algorithm therefore may not converge to the actual fixed point of the mean-field updates . This is because Baqué et al . ( 2016 ) used the prox1f proximal operator ( described below ) , whereas we derive the proxαf operator to guarantee global convergence when doing mean-field updates in parallel . What ’ s more , Baqué et al . ( 2016 ) focused only on inference over prescribed potentials , and not on training the ( fully parameterized ) potentials as we do here . Lê-Huu & Alahari ( 2021 ) brought up a generalized Frank-Wolfe based framework for mean-field updates which include the methods proposed by Baqué et al . ( 2016 ) ; Krähenbühl & Koltun ( 2013 ) . Their results only guarantee global convergence to a local optimal . 3 MONOTONE DEEP BOLTZMANN MACHINES AND APPROXIMATE INFERENCE . In this section , we present the main technical contributions of this work . We begin by presenting a parameterization of the pairwise potential in a Boltzmann machine that guarantees the monotonicity condition . We then illustrate the connection between a ( joint ) mean-field inference fixed point and the fixed point of our monotone Boltzmann machine and discuss how deep structured networks can be implemented in this form practically ; this establishes that , under the monotonicity conditions on Φ , there exists a unique globally-optimal mean-field fixed point . Finally , we present an efficient parallel method for computing this mean-field fixed point , again motivated by the machinery of monotone DEQs and operator splitting methods . 3.1 A MONOTONE PARAMETERIZATION OF GENERAL BOLTZMANN MACHINES . In this section , we show how to parameterize our probabilistic model in a way that the pairwise potentials satisfy I − Φ ⪰ mI , which will be used later to show the existence of a unique meanfield fixed point . Additionally , since Φ defines a graphical model that has no self-loop , we further require Φ to be a block hollow matrix ( that is , the ki × ki diagonal blocks corresponding to each variable must be zero ) . While both these conditions on Φ are convex constraints , in practice it would be extremely difficult to project a generic set of weights onto this constraint set under an ordinary parameterization of the network . Thus , we instead advocate for a non-convex parameterization of the network weights , but one which guarantees that the monotonicity condition is always satisfied , without any constraint on the weights in the parameterization . Specifically , define the block matrix A = [ A1 A2 · · · An ] ( 2 ) with Ai ∈ Rd×ki matrices for each variables , and where d can be some arbitrarily chosen dimension . Then let Âi be a spectrally-normalized version of Ai Âi = Ai ·min { √ 1−m/∥Ai∥2 , 1 } ( 3 ) i.e. , a version of Ai normalized such that its largest singular value is at most √ 1−m ( note that we can compute the spectral norm of Ai as ∥Ai∥2 = ∥ATi Ai∥ 1/2 2 , which involves computing the singular values of only a ki × ki matrix , and thus is very fast in practice ) . We define the Â matrix analogously as the block version of these normalized matrices . Then we propose to parameterize Φ as Φ = blkdiag ( ÂT Â ) − ÂT Â ( 4 ) where blkdiag denotes the block-diagonal portion of the matrix along the ki×ki block . Put another way , this parameterizes Φ as Φij = { −ÂTi Âj if i ̸= j , 0 if i = j . ( 5 ) As the following simple theorem shows , this parameterization guarantees both hollowness of the Φ matrix and monotonicity of I −Φ , for any value of the A matrix . Theorem 3.1 . For any choice of parameters A , under the parametrization equation 4 above , we have that 1 ) Φii = 0 for all i = 1 , . . . , n , and 2 ) I −Φ ⪰ mI . Proof . Block hollowness of the matrix follows immediately from construction . To establish monotonicity , note that I −Φ ⪰ mI ⇐⇒ I + ÂT Â− blkdiag ( ÂT Â ) ⪰ mI ⇐= I − blkdiag ( ÂT Â ) ⪰ mI ⇐⇒ I − ÂTi Âi ⪰ mI , ∀i ⇐⇒ ∥Âi∥2 ≤ √ 1−m , ∀i . ( 6 ) This last property always holds by the construction of Âi .

Summary: In this paper the authors propose a restricted parameterization of the Boltzmann machine that guarantees that for any set of observations, the mean field objective has a single global optimum. Furthermore, that global optimum can be provably achieved using damped parallel mean-field updates, which make inference efficient. To turn inference into learning, the model is treated as a supervised learning model: some of its variables are considered to be observed inputs and some of its variables are considered to be target outputs (known at test time). The usual, marginal cross-entropy loss is the optimization target for learning.