Monotone deep Boltzmann machines
Authors: Zhili Feng
Ezra Winston
J Zico Kolter
Venue: Accepted by TMLR ()

Abstract: Deep Boltzmann machines (DBMs), one of the first ``deep'' learning methods ever studied, are multi-layered probabilistic models governed by a pairwise energy function that describes the likelihood of all variables/nodes in the network. In practice, DBMs are often constrained, i.e., via the \emph{restricted} Boltzmann machine (RBM) architecture (which does not permit intra-layer connections), in order to allow for more efficient inference.  In this work, we revisit the generic DBM approach, and ask the question: are there other possible restrictions to their design that would enable efficient (approximate) inference?  In particular, we develop a new class of restricted model, the monotone DBM, which allows for arbitrary self-connection in each layer, but restricts the \emph{weights} in a manner that guarantees the existence and global uniqueness of a mean-field fixed point. To do this, we leverage tools from the recently-proposed monotone Deep Equilibrium model and show that a particular choice of activation results in a fixed-point iteration that gives a variational mean-field solution.  While this approach is still largely conceptual, it is the first architecture that allows for efficient approximate inference in fully-general weight structures for DBMs.  We apply this approach to simple deep convolutional Boltzmann architectures and demonstrate that it allows for tasks such as the joint completion and classification of images, within a single deep probabilistic setting, while avoiding the pitfalls of mean-field inference in traditional RBMs.

Published in Transactions on Machine Learning Research (04/2023)
Monotone deep Boltzmann machines
Zhili Feng
shilifSandrew.. cmnu, eduu
Machine Learning Department
Carnegie Mellon University
Ezra Winston
ewinston@cs. cmnu. edu
Machine Learning Department
Carnegie Mellon University
J. Zico Kolter
akolter@cs.cmu.com
Computer Science Department Carnegie Mellon University Bosch Center for Al
Reviewed on OpenReview: https: // openreview. net/ forum? id= SgIKköryPr
Abstract
Deep Boltzmann machines (DBMs), one of the first "deep" learning methods ever studied, are multi-layerod probabilistic models governed by pairwise energy function that deseribes the likelihood of all variables nodes in the network. In practice, DBMs often constrained, ie., via the restricted Boltzmann machine (RBM) architecture (which does not permit intra- layer connections), in order to allow for more efficient inference. In this work, revisit the generic DBM approach, and ask the question: are there other possible restrictions to their design that would enable efficient. (approximale) inference? In partieular, we develop a new class of restricted model, the monotone DBM, which allows for arbitrary confeconnecton in each layer, but restricts the weights in a manner that guarantees the existence and global uniqueness of a mean-field fixed point. To do this, we leverage tools from the recently- proposed monotone Deep Equilibrium model and show that particular choice of activation results in fixed-point iteration that gives variational mean-field solution. While this approach is still largely conceptual, it is the first architecture that allows for efficient ap- proximate inference in fully-general weight structures for DBMs. We apply this approach to simple deep convolutional Boltzmann architectures and demonstrate that it allows for tasks such as the joint completion and classification of images, within single deep probabilistic setting, while avoiding the pitfalls of mean-fiele inference in traditional RBMs.
1 Introduction
This paper considers (deep) Boltzmann machines (DBMs), which are pairwise energy-based probabilistic models given by a joint distribution over variables x with density
p(x) x exp
Σ (i,)E
I 4
Dij
+
Σ
6,
(1)
where each Fitn denotes a discrete random variable over k; possible values, represented as one-hot encoding 24 E {0, k;; E denotes the set of edges in the model; 4, E x represents pairwise potentials; and b A represents unary potentials. Depending on the context, these models are typically referred pairwise Markov random fields (MRFs) (Koller Friedman 2009), or (potentially deep) Boltzmann machines Goodfellow et al.) 2016 Salakhutdinov & Hinton
2009 Hinton 2002). Tn the above setting, each x; may

---

Published in Transactions on Machine Learning Research (04/2023)
Restricted Boltzmann machine
Restricted deep Boltzmann machine
General Boltzmann machine
X/y
Xby
Xn
4 jib
Xhr
Xhi
S ho
ho
X
X
Figure 1: Neural network topology of different Boltzmann machines. The general case is complete graph (red dashed lines are subset of edges that are in BM but not RBM). Our proposed parameterization is form of general Boltzmann machine.
represent an observed or unobserved value, and there can be substantial structure within the variables; for instance, the collection of variables may (and indeed will, in the main settings we consider in this paper) consist of several different "layers" in joint convolutional structure, leading to the deep convolutional Boltzmann machine (Norouzi et al. 2009b.
Boltzmann machines were some of the first. "deep" networks ever studied Ackley et al. (1985) However, in modern deep-learning practice, general-form DBMs have largely gone unused, in favor of restricted Boltzmann machines (RBMs). These are DBMs that avoid any connections within single layer of the model and thus lead themselves more efficient. block-based approximale inference methods.
In this paper, we revisit the general framework of generic DBM, and ask the question: are there any other restrictions (besides avoiding intra-layer connections). that would also allow for efficient approximate inference methods? To answer this question, we propose a new class of general DBMs, the monolone deep Boltzmann machine (mDBM); unlike RBMs, these networks can have dense intra-layer connections but are parameterized in a manner that constrains the weights so as to still guarantee an efficient inference procedure. Specifically, in these networks, we show that there is a unique and globally optimal fixed point of variational mean-field inference; this contrasts with traditional probabilistic models where mean-field inference may lead to multiple diffcrent local optima. To accomplish this goal, we leverage recent work monotone Deep Equilibirum (monDEQ) models (Winston & Kolter) 2020), and show that particular choice of aclivation function leads to fixed point iteration equivalent to (damped) parallel mean-field updates. Such fixed point iterations require the development of a new proximal operator method, for which we derive a highly efficient GPU-based implementation.
Our method also relates closely with previous works on convergent mean-field inference in Markov random fields (MRFs) (Krähenbühl & Koltun 2013 Baqué et al.) 2016 Lê-Huu & Alahari 2021); but these ap- proaches either require stronger conditions the network or fail to converge to the true mean-field fixed point, and generally have only been considered on standard "single-layer" MRFs. Our approach can be viewed as a combined model parameterization and (properly damped) mean-field inference procedure, such that the resulting iteration is guaranteed to converge to a unique optimal mcan-field fixed point when run in parallel over all variables.
Although the approach still largely conceptual, we show for the first time that one can learn and perform inference in structured multi-layer Boltzmann machines which contain intra-layer connections. For example, perform both learning and inference for deep convolutional, multi-resolution Boltzmann machine, and apply the network to model MNIST and CIFAR-10 pixels and their classes conditioned on partially observed images. Such joint probabilistic modeling allows us to simultaneously impule missing pixels and predict the class. While these are naturally small-scale tasks, we emphasize that performing joint probabilistic inference over a complete model of this type a relatively high-dimensional task as far as traditional mean-field inference is concerned. We compare our approach (block structured) mean-field inference in classical RBMs, showing substantial improvement. in these estimates, and also compare to alternative mean-field
2

---

Published in Transactions on Machine Learning Research (04/2023)
inference approaches. Although in initial phases, the work hints at potential new directions for Boltzmann machines involving very different types of restrictions than what has typically been considered in deep learning.
2 Background and related work
This paper builds upon three main avenucs of work: 1) doep equilibrium models, especially one of their convergent version, the monotone DEQ; 2) the broad topic of energy-based deep model and Boltzmann machines in particular; and 3) work on concave potentials and parallel methods for mean-field inference. We discuss each of these below.
Equilibrium models and their provable convergence The DEQ model was first proposed by Bai et al. (2019). Based on the observation that a neural network g = (Wq:+Uu+t) with input injection æ usually converges to a fixed point, they modcled an effectively infinite-depth network with input injection directly via its fixed point: y² = (W HWWW"++U' +Ux + b). Its backpropagation is done through the implicit function theorem and only requires constant memory. Bai et al. (2020) also showed that the multiscale DEQ models achieve near state-of-the-art performances on many large-scale tasks. Winsson & Kolter (2020) later presented a parametrization of the DEQ (denoted monDEQ) that guarantees provable convergence to a unique fixed point, using monotone operator theory. Specifically, they parameterize W in a way that J- W > m7 (called m-strongly monotone) is always satisfied during training for some m > O; they convert nonlinearities into proximal operators (which include ReLUU, tanh, elc.), and show that using existing splitting methods like forward-backward backward and Peaceman- Peaceman-Rachford can provably find the unique fixed point. Other notable related implicit model works include Revay et al. (2020), which enforces Lipschitz constraints on DEQs; El Ghaoui et al. (2021) provides thorough introduction to implicit models and their well- sell-posedness. [Tsuchida & Ong
2022 also solves graphical model problems using DEQs, focusing on principal component analysis.
Markov random field (MRF) and its variants MRF is a form of enrgy-based models, which model joint probabilities of the form (0 exp (-Fo(z)) /Zg for an energy function Eg. A common type of MRF is the Boltzmann machine, the most successful variant of which is the restricted Boltzmann machines (RBM) (Hinton) 2002) and its deep (umult--layer) variant (Salakhutdiinov & Hinton (2009). Particularly, RBMs define (v, -T 6 h x Wh, where A = (W, a, b), is the set of visible variables, and h is the set of latent variables. It is usually trained using the contrastivediieerenne algorithm, and its inference can be done efficiently by block mean-field approximation. However, a particular restriction of RBMs is that there can be no intra-layer connections, that is, each variable in v (resp. h) is independent conditioned on h (resp. v). A deep RBM allows different layers of hidden nodes, but there cannot be intra-layer connections. By contrast, our formulation allows intra-layer connections and is therefore more expressive in this respect. See Figure] 1 for the network topology of RBM, deep RBM, and general BM (we also use the term general deep BM interchangeably to emphasize the existence of deep structure). [Wu et al.) (2016) proposed a deep parameterization of MRF, but their setting only considers grid of hidden variables h, and the connections among hidden units are restricted to the neighboring nodes. Therefore, it is a special case of our parameterization (although their learning algorithm is orthogonal to ours). Numerous works also try to combine deep neural networks with conditional random fields (CRF)) (Krähenbühl & Koltun 2013; Zheng et al. 2015 Schwartz et al.) 2017) These models cither train a pro-determined kernel as an RNN or use neural networks for producing either inputs or parameters of their CRFs.
Parallel and convergent mean-field It is well-known that mean-field updates converge locally using coordinate ascent algorithm (Blei et al. 2017). However, local convergence is only guaranteed if the update is applied sequentially. Nonetheless, several works have proposed techniques to parallelize updates, Krähenbühl & Koltun (2013) proposed a concave-convex procedure (CCCP) to minimize the KL divergence between the true distribution and the mean-field variational family. To achieve efficient inference, they use a concave approximation to the pairwise kernel, and their fast update rule only converges if the kernel function concave. Later, Baqué et al. (2016) derived similar parallel damped forward iteration ours that provably converges without the concave potential constraint. However, unlike our approach, they do not use parameterization that ensures global mean-field optimum, and their algorithm therefore may
3

---

Published in Transactions on Machine Learning Research (04/2023)
not converge to the actual fixed point of the mean-field updates. This is because Baqué al (2016) used the prox proximal operator (described below), whereas we derive the proxý operator to guarantee global convergence when doing mean-field updates in parallel. What's more, [Bqqué et al.) (2016) focused only on
inference over prescribed potentials, and not on training the (fully paramcterized) potentials as we do here. Lê-Huu & Alahari (2021) brought up a generalized Frank- Wolfe based framework for mean-field updates
which include the methods proposed by Baqqé et al. (2016): Krähenbühl & Koltun (2013). Their results only guarantee global convergence local optimal.
3 Monotone deep Boltzmann machines and approximate inference
In this section, we present the main technical contributions of this work. We begin by presenting a param- eterization of the pairwise potential in Boltzmann machine that guarantees the monotonicity condition. We then illustrate the connection between a (joint) mean-field inference fixed point and the fixed point of our monotone Boltzmann machine (mDBM) and discuss how deep structured networks can be implemented in this form practically; this establishes that, under the monotonicity conditions on 4, there exists a unique globally-optimal mean-field fixed point. Finally, we present an efficient parallel method for computing this mean-field fixed point, again motivated by the machinery of monotone DEQs and operator splitting methods.
3.1 A monotone parameterization of general Boltzmann machines
In this section, we show how to parameterize our probabilistic model in a way that the pairwise potentials satisly --$ $ > ml, which will be used later to show the existence unique mean-field fixed point. Recall that & defines the interaction between random variables in the graph. In particular, for random variables Ti E RR*, Iz R*, we have Pig E R& Additionally, since & defines graphical model that has no self-loop, we further require & to be block hollow matrix (that is, the k, x k2 diagonal blocks corresponding to each variable must be zero). While both these conditions on + are convex constraints, in practice it would be extremely difficult to project a generic set of weights onto this constraint set under an ordinary parameterization of the network.
Thus, we instead advocate for a non-conver parameterization of the network weights, but one which guar- antecs that the monotonicity condition always satisfied, without any constraint on the weights in the parameterization. Specifically, define the block matrix
A A= A1 A2
A,
with A, Raxk, matrices for each variables, and where can be some arbitrarily chosen dimension. Then let A, be spectrally-non normmallyy-oomalizeed version of A;
Å A, min /1 A.l/,, 1}
i.., a version of A; normalized such that its largest singular value is at most vI- (note that we compute the spectral norm of A, as || = ||A A, which involves computing the singular values of only a K; x K; matrix, and thus is very fast in practice). We define the À matrix analogously as the block version of these normalized matrices.
Then we propose parameterize as
blkdiag( AT A) À
(3)
where bikdiag denotes the block-diagonal portion of the matrix along the h; k, block. Put another way, this parameterizes 4
AT A
if
(4)
0
i-
As the following simple theorem shows, this parameterization guarantees both hollowness of the & matrix and monotonicity of I 4, for any value of the A matrix.
4

---

For Published in Transactions on Machine Learning Research (04/2023)
Theorem 3.1. For any choice of parameters A, under the paramctrization equation 3 above, have that 1) a 0 for all i 1..i=,...... and 2) T + = ml.
Proof. Block hollowness of the matrix follows immediately from construction. To establish monotonicity, note that
I- + 2 ml I+ A Á blkdiag( A A) = ml
= I - blkdiag( (aatiiag(4* AA A) = ml = 1 A$ A ml, Wi
(5)
| vI m, Vi.
This last property always holds by construction of A..
3.2 Mean-field inference as a monotone DEQ
In this section, we formally present how to formulate the mean-field inference as a DEQ update. Recall from before that we are modeling a distribution of the form Equation ① We are interested in approximating the conditional distribution o(x//xx), where o and denote the observed and hidden variables respectively, with a factored distribution ((x)) Ilee as (mz)) Here, the standard mean-field updates (which minimize the KL divergence between (Xx)) and p(xx| over the single distribution (xx))) are given by the following equation,
(x(x)) == softmax
Σ
(x)))(;;)) + bi
HIJJEE
where overloading notation slightly, let (xr,) denote one-hot encoding of the observed value for any E o (see e.g.. Koller & Friedman (2009) for full derivation).
The essence of the above updates a characterization of the joint fixed point to mean-field inference. simplicity of notation, defining
[q) (
(2(222 ...]².
We see that qu is a joint fixed point of all the mean-field updates if and only if
q = softmax (n + PhoXo + b,)
(6)
where x, analogously denotes the stacked one-hot encoding of the observed variables.
We briefly recall the monotone DEQ framework of Winston & Kolter (2020). Given input vector x, a monotone DEQ computes the fixed point x =xxx) that satisfies the equilibrium equation =x(x) = (W** (x*(x) + Ux + b). Then 1) a given by a proximal operator (x) (r)x)(x)) for some convex closed proper (CCP) f, and if have the monotonicity condition W ž ml (in the positive semidefinite sense) for some 0, then for any there exists unique fixed point 2 =x(x), which can be computed through standard operator splitting methods, such
splitting.
We now state our main claim of this subsection, that under certain conditions the mean-field fixed point can be viewed as the fixed point of an analogous DEQ. This is formalized in the following proposition.
Proposition 3.1. Suppose that the pairwise kernel & satisfies - + Z ml for m > 0. Then the mean-field fized point
q softnax (Phnhh DroXo bn)
(7)
corresponds to the fized point of a monotone DEQ model. Specifically, this implies that for any X., there exisls unique, globalty-optimal fixed point of the mean-field distribution qr.
A proximal operator is defined by prox (x) aigmin, 1 3|< =|I2 az)
Trechnicaly speaking, only need - a.. mī, since want this hold any choice of h, we need the condition apply to the entire matrix.
5

---

Published in Transactions on Machine Learning Research (04/2023)
91
42
43
A4
As1
(i(Ti)
Figure 2: Illustration of possible deep convolutional Boltzmann machine, where the monotonicity structure can still be enforced.
Proof. As the monotonicity condition of the monotone DEQ is assumed in the proposition, the proof of the proposition rests entirely showing that the softmax operator is given by prox} for some CCP f. Specifically, as shown in Krähenbühh Koltun (2013) this is the case for
f(z) Σ
24 log 2 2112112 +I 2 = 2=1,220
(8)
ie., the restriction of the entropy minus squared norm to the simplex (note that even though we are sub- tracting a squared norm term it is straightforward to show that this function is convex since the second derivatives are given by 1/2, - 1, which is always non-negative over its domain).
3.3 Practical considerations when modeling mDBMs
The construction in Section 3.1 guaranlees monotonicity of the resulting pairwise probabilistic model. How- ever, instantiating the model in practice, where the variables represent hidden units a deep architecture (i.c., representing multi-channel image tensors with pairwise potentials defined by convolutional operators), requires substantial subtlety and care in implementation. In this setting, we do not want actually rep- resent A explicitly, but rather determine method for multiplying Av and ATv for some vector v (as we see in Section 3.2, this is all that is required for the parallel mean-field inference method propose). This means that certain blocks of A are typically parameterized as convolutional layers, with convolution and transposed convolution operators as the main units of computation.
More specifically, we typically want to partition the full set of hidden units into some distinct sets
91
(9)
where e.g., qs would be best represented as a height x width x groups x cardinality tensor (i.c., collection of multiple hidden units corresponding different locations in a typical deep network hidden layer). Note that here qi is not the same as (Ix), but rather the collection of many different individual variables. These qe terms can be related to each other via different operators, and natural manner of parameterizing A, in this dense operators. To represent the pairwise interactions, we can create similarly-factoree matrix A, e.g,, one of the form
case, is as an interconnected set of convolutional
An
0
A21
A
(10)
Aκι
AкK
where c.g., Ai is (possibly strided) convolution mapping between the tensors representing q, and In this case, we emphasize that Ai is not the kernel matrix that one "slides" along the variables. Instead, A;j is the linear mapping as we write the convolution as a matrix-mant matrix multiplication. For example, a 2D convolution with stride 1 can be expressod as a doubly block circulant matrix (the case is more complicated when different striding is allowed). This parametrization effectively a general Boltzmann machine, since each random variable in Equation
(9) can interact with any other variables except for itself. Varying A,j.

---

Published in Transactions on Machine Learning Research (04/2023)
the formulation in Equation (10) is rich enough for any type of architecture including convolutions, fully-
connected layers, and skip-comeectonss, etc.
An illustration of one possible network structure is shown in Figure 2 To give a preliminary introduction to our implementation, let us denote the convolution filter as F, and its corresponding matrix form as A. While it possibly simpler to directly compute A T Aq, usually has very high dimensions even if F small. Instead, our implementation computes CONVTRANSPOSEL and padding. The block diagonal element of A A has smaller dimension and can be computed directly 1 X convolution. The precise details of how one computes the block diagonal elements of AA' A, and how one normalizes the proper diagonal blocks (which, we emphasize, still just requires computing the singular values of matrices whose size is the cardinality of a single gi(x)) are somewhat involved, so we defer a complete description to the Appendix (and accompanying code). The larger takeaway message, though, is that it is possible parameterize compler convolutional multi-scale Boltzmann machines, all while ensuring monotonicity.
CoNv(F, q)), modulo using the correct striding
3.4 Efficient parallel solving for the mean-fiele fixed point
Although the monotonicity of t guarantees the existence of a unique solution, does not necessarily guar- antee that the simple iteration
= softmax (P P (t-1) + Phowt + bn)
(11)
will converge to this solution. Instead, to guarantce convergence, one ncods to apply the damped iteration (see, e.g. (Winston & Kolter ))20
= prox (11-a)gh
(1-1) + a a(Pingh (1-1) + Prowo + bb))
(12)
The damped forward-backward iteration converges linearly to the unique fixed point if a S 2m/ T,2, assuming I T-a + is m-strongly monotone and L-Lipschitz (Ryu & Boyd, 2016). Crucially, this update can be formed in parallel over all the variables in the network: we do not require coordinate descent approach as is typically needed by mean-field inference.
The key issue, though is that while prox (x) = softmax(x)) for f defined as in Equation 8 general, this does not hold for a # 1. Indeed, for a * 1, there is no closed-form solution the proximal operation, and computing the solution is substantially more involved. Specifically, computing this proximal operator involves solving the optimization problem
(x) arg min 2
1 |
211 +
Σ 24 log 2,
a 2 11 s.t Σ 1==1, =1 =20 O.
(13)
The following theorem, proved in the Appendix, characterizes the solution to this problem for a E (0, 1) (although is also possible to compute solutions for a > 1, this is not nceded in practice, as it corresponds to a "negatively damped" update, and is typically better to simply use the softmax update in such cases).
Theorem 3.2. Given f as defined in Equation is given by
S a E (0, 1), and x E R the prozimal operator prox} (x)
Tt xi=a+ 1 exp
prox)(=);= 1- W
a
where AE R. the unique solution chosen to ensure that the resulting E, proxf = 1, and where W(-) is the principal branch of the Lambert W function.
In practice, however, this is not the most numerically stable method for computing the proximal operator, especially for small a, owing to the large term inside the exponential. Computing the proximal operation efficiently is somewhat involved, though briefly, define the alternative function
g(u) = log
a W
1 (L-a a exp
- -1)
(14)
1 a

---

Published in Transactions on Machine Learning Research (04/2023)
7
2
4 4 7
4
A
2
1
0
4
1
4
9
7
2.
1
0
4
A
7
2
4
A
7
9
0
5
5 9
0
9
0
5
5
9
0
1
9
0
1
5
7
0
5
4
7
4
5
9 7 0
8
9
6
6
7
8
4
7
40
5
9
7
7
4
5
4
4
4
0
4
0
7
4
0
3
4
30
7
4
0
3
1
3
2
2
2-
3
4
7
2 7
2
3 7
4
2
7
1
3
6
7
2
7
(a) Test data with 60% pix-
(b) Original image
(c) Imputation with 60%
(d) Imputation with 60%
randomly masked
pixels randomly masked uss- pixels randomly masked us- ing mDBM
ing RBM
Figure 3: MNIST pixel imputation using mDBM (bottom left) and deep RBM (bottom right), where the RBM test results are generated using mean-field inference instead of Gibbs sampling.
and show how to directly compute g using Halley's method (note that Halley's 's method is also the preferred manner lo computing the Lambert W function itself numerically (Corless et al. 1996)). It updates In+1 In -(an and enjoys cubie convergence when the initial guess is close enough to the root. Finding the prox operator then requires that we find such that Σ exp(g(æ; + A)) = This can be done via con-dimmnsional) root finding with Newton's method, which is guaranteed to always find a solution here, owing to the fact that this function is convex monotonic for À (-o0, 1). We further compute the gradients of the g function and of the proximal operator itself via implicit differentiation (i.e., can do it analytically without requiring unrolling the Newton or Halley iteration). We describe the details in the appendix and include an efficient PyTorrh function implementation in the supplementary material.
(f)
Comparison to Winston & Kolter (2020) Although this work uses the same monotonicity constraint as in Winston & Kolter (2020), our result. further requires the linear module to be hollow, and extend their work to the softmax nonlinear operator as well. These extensions introduce significant complications, but also enable 22020 cannot.
interpret our network as probabilistic model, while the network Winston & Kolter
3.5 Training considerations
Finally, we discuss approaches for training mDBMs, exploiting their efficient approach to mean-field inference. Probabilistic models are typically trained via approximate likelihood maximization, and since the mean- field approximation is based upon a particular likelihood approximation, it may seem most natural to use this same approximation to train parameters. In practice, however, this often a suboptimal approach. Specifically, because our forward inference procedure ultimately uses mean-field inference, it is better to train the model directly to output the correct marginals, when running this mean-field procedure. This is known as marginal-based loss (Domke 2013). In the context of mDBMs, this procedure has a particularly convenient form, it corresponds roughly to the "typical" training of DFQ.
In more detail, suppose we given sample E X (i.e,, at training time the entire sample is given), along with specification of the "observed" and "hidden" sets, and h respectively. Note that the choice of observed and hidden sets is potentially up to the algorithm designer, and can effectively allow one to train our model in a "self-supervised" fashion, where the goal is to predict some unobserved components from others. In practice, however, one typically wants to design hidden and observed portions congruent with the eventual use of the model: e.g., if one using the model for classification, then training time it makes sense for the label to be "hidden" and the input to be "observed."
Given this sample, we first solve the mean-field inference problem to find g (xx) such that
q softmax ( + @ + by).
(15)
For this sample, we know that the true value of the hidden states is given by x Thus, we can apply some loss function Eggi xx) between the prediction and true value, and update parameters of the model
8

---

Published in Transactions on Machine Learning Research (04/2023)
0 (A using their gradients
dll (1
0E(q%, x
Ōq..
al(q xh.) (I I
dg(g%)
0g(g%)
(16)
00
dqq.
00
0q
0qh
00
with
( = prox3 ((1 - a. ar PPnn + Pio + b₁))
and where the last equality comes from the standard application of the implicit function theorem as typical in DEQs or monotone DEQs. The key to gradient computation is noticing that in Equation (16), we can rearrange:
dllgh,
(1- Og(gh)
3g(gh) aE( mn
u 0q
0q
0q%
0qh
which is just another fixed-point problem. Here all the partial derivatives can be handled by auto- differentiation (with the correct backward hook for proxf), and the details exactly mirror that of Winston & Kolter (2020), also see Algorithm 2
As final note, we also mention that owing to the restricted range of weights allowed by the monotoniety constraint, the actual output marginals q:(x.)) often more uniform in distribution than desired. Thus, typically apply the loss to scaled marginal
( x go (x:)"
(17)
where E R, is a variable-dependent learnable temperature parameter. Importantly, we emphasize that this only done after convergence to the mean-field solution, and thus only applies to the marginals to which we apply a loss: the actual internal iterations of mean-field cannot have such a scaling, as would violate the monotonicity condition.
Algorithm 1 FORWARDITERATION
Require: Observed RV æo; parameters 4 b, damp parameter (0, Find the fixed point of to
dn (0)
prox? (11 ((1 (-11)) af a(Fngg (1-1) + @noto + bb))
Algorithm 2 BACKWARDITERATION
Require: Loss function e, fixed point hh true value æy parameters 0. Find the fixed point u² to
dg(g%)
Ol(g), xx))
0qh
0q
Compute the final Jacobian-vector product as
dc (1( xn)
3g(g%))
2
00
00
A simple overview of the algorithm is demonstrated Algorithm X The task to jointly predict the image label and fill the tol-half of the image given the bottom-half For inference, the process is almost the same training, except don't update the parameters.

---

Published in Transactions on Machine Learning Research (04/2023)
Algorithm 3 TRAINING
Require:
Damp parameter (0, 1), neural network parameters Φ, weight classification loss w. for each epoch do
for (p. u) in data do
Let 2;, = (p.. y) where Pı. is the top-half of the image p and y is the label. Let Zo be the bottom-half of the image Po- Li. FFFRRAADDTTRRTATTNNNG
Φb b, az), notice that is not revealed to the network here. Get the estimated top half of the image and label {pr, 0) = qh.
Calculate imputation loss and classification loss (1 - w)e, (ph. Ph)) + wlc(0.
Update 4, b using Algorithm
end for end for
(a) Test data has 50% pixels randomly masked
(b) Imputed pixel inference (without injection labels)
(c) Original image
(d) Imputation with deep
RBM.
Figure 4: CIFAR-10 pixel imputation using mDBM and deep RBM
4 Experimental evaluation
As a proof of concept, evaluate our proposed mDBM the MNIST and CIFAR-10 dalasets. We demon- strate how to jointly model missing pixels and class labels conditioned on only a subset of observed pixels. On MNIST, we compare mDBM to mean-field inference in a traditional deep RBM. Despite being small-scale tasks, the goal here is to demonstrate joint inference and learning over what still a reasonably-sizee joint model, considering the number of hidden units. heleth the current experiments are admittedly largely demonstration of the proposed method rather than full accounting of its performance.
We also show how our mean-field inference method compares to those proposed in prior works. On the joint imputation and classification task, we train models using our updates and the updates proposed by Krähenbühl & Koltun (2133) and Baqué et al. (2016), and perform mean-field inference in each model using all three update mothods, with and without the monotonicity constraint.
mDBM and deep RBM on MNIST For the joint imputation and classification task, we randomly mask each pixel independently with probability 60%, such that in expectation only 40% of the pixels are observed. The original MNIST dataset has one channel representing the gray-scale intensity, ranging between and 1. We adopt the strategy of Van Oord et al. (2016) to convert this continuous distribution to a discrete one. We bin the intensity evenly into 4 categories (o....,.) .....,,),,... ,,,, and for each channel use a one-hot encoding of the category so that the input data has shape x 28. We remark that the number of categorics chosen arbitrarily and can be any integer. Additional details are given in the appendix.
The mDBM and deep RBM trained on the joint imputation and classification task obtain test classification accuracy of 92.95% and 64.23%, respectively. Pixel imputation is shown in Figure 3 We see that the deep RBM is not able to impule the missing pixels well, while the mDBM can. Importantly, we note however that for an apples-to-apples comparison, the test results in the RBM are generated using mean-field inference. The image imputation of RBM runs block mean-field updates of 1000 steps and the classification runs 2 steps, and increasing number of iterations does not improve test performance significantly. The RBM also admits efficient Gibbs sampling, which performs much better and is detailed in the appendix.
10

---

Published in Transactions on Machine Learning Research (04/2023)
1 I U
7
2
4
4
1
4
9
7
2
1
6
4
1
4
9
5
9
0
6
9
0
1
5
5
9
0
6
9
0
5
9
7
3
4
4
6
4
5
9
7
8
4 4
9
6
6
5
4
0
7
4
0
1
3
4
0
7
0
2
3
U
7
2
7
1
3
4
7
2
7
2
MNIST and CIFAR10 pixel imputation using mDBM, when only the top half is shown to the model. Left: observed images; middle: imputation results; right: original images.
Figure 5:
Table 1: Squared l error (standard deviation) for MNIST image imputation. Images are observed 20%, 40%, 60%, 80% respectively. RBM outputs are bucketized into 10 bins. The errors averaged over the whole dataset and the number of bins. The experiments are executed 5 times with independent random masks and the same models. Standard deviations are calculated across runs.
Observation
0.2
0.4
0.6
0.8
Method
mDBM
53.310 (0.0776)
23.330 (0.0201)
13.140 (0.0234)
5.936 (0.0102)
RBM
53.086 (0.0556)
36.596 (0.0417)
22.746 (0.0234)
10.564 (0.0186)
We report the image imputation & loss on MNIST in Table 1 We randomly mask = (..2, 0.4, 0.8} portion of the inputs. For each p, the experiments are conducted 5 times where cach run independently chooses the random mask. The model is trained to impute images given 40% pixels and fixed throughout the experiments. Since the RBMs model Bernoulli random variables whereas mDBMs model distributions over set of one-hot variables, the difference between bucketized imputations and the original images is computed. The norms are then divided by the number of bins and averaged over the whole MNIST dalaset. In this way, we gel {/1,...,,45] where each μu is the average l reconstruction error over the whole dataset, and the standard deviations are calculated over (ss. Our proposed method has a clear advantage over RBMs.
"bucketize" f the outputs of RBMs into bins of size 10. The E norm of
We additionally evaluate mDBM on task in which random 14x 14 patches are masked, To obtain good per- formance this task requires lifting the monotonicity constraint; find that mDBM converges regardless (see appendix). mDBM can also extrapolate reasonably well, see Figure
mDBM and deep RBM on CIFAR-10 We evaluate mDBM on an analogous task of image pixel imputation and label prediction on CIFAR-10. Model architecture and training details are given in the appendix. With 50% of the pixels observed, the model obtains 58% test accuracy and can impute the missing pixels effectively (sce Figure 4 The bascline deep RBM trained impute the missing pixels using CD-1 with number of neurons 3072-500-100. The imputation error is reported in Table 2 and the experiments conducted in the same fashion those MNIST. Contrary to grayscale MNIST, RBM outputs are bucketized into 10 bins for each of the RGB channels on CIFAR-10, mDBMs also take bucketized images as inputs where each of the RGB channels is bucketized into 10 bins.
3hhat is, the number of bins is and the RBM outputs probability p. the bucketized output is 0 p < 0.5 and otherwise.
11

---

Published in Transactions on Machine Learning Research (04/2023)
Table Squared & error (standard deviation) for CIFAR10 image imputation. Images are observed 20%, 40%, 60%, 80% respectively. RBM outputs are bucketized into 10 bins for each of the RGB chan- nels. mDBMs also take bucketized images as imputs where each of the RGB channels is bucketized into 10 bins. The errors averaged over the whole dataset and the number of bins. The experiments executed 5 times with independent random masks and the same models. Standard deviations are calculated across 5 runs.
Observation
0.2
0.4
0.6
0.8
Method
mDBM
77.439 (0.0281)
48.496
(0.0166)
29.749 (0.0241)
14.077 (0.01-13)
RBM
139.375 (0.0231)
103.615 (0.0315)
68.845 (0.0219)
34.339 (0.0291)
Table 3: Relative update residual when monotonicity is enforced.
Humber of
Inference
Krähenbühl
Baqué
mDBM
Train
Krähenbühl
0.0004
0.0061
0.0024
0.28
Baqué
1.250
0.0059
0.0024
mDBM
1.144
0.0057
0.0017
Figure 6: Convergence speed of infer- ence methods on a model trained with Krähenbühl's updates.
Comparison of inference methods We conduct several experiments comparing our mean-field inference method to those proposed by Krähenbühl & Koltun (2013) and Baqué ct al. (0166), denoted as Krähenbüihl's and Baqué's respectively. While full description of these melhods and the experiments left to the appendix, we highlight some of the key findings here. We train models using the three different update methods: ours, Krähenbühl's and Baqué's; we then perform inference using all three methods well. A comparison to the regularized Frank- Wolfe method raised by Lě-Huu & Alahari (2021) can be found in the appendix.
after 100 steps of each inference method each model. We observe that Krähenbühl's method diverges when the model was not trained using the corresponding updates, whereas Baqué's and ours converge on all three models, with our method converging more quickly. The improved convergence speed can also be seen in Figure G). However, note that Baqué's method is not guaranteed to converge to the true mean-field fixed-point. As we show in the appendix (Figure I1cl, on an untrained model our method converges to the true point while Baqué's does not.
Table shows the relative update residuals |/ (-/ Lh (4) 1//l (IIl
Future directions It is extremely useful to consider fundamentally different restrictions as have been applied to meanfield inference and graphical models in the past, and our work can lead to a number of interesting directions: (1) Theorem 3.1 is only a sufficient but not necessary condition for monotonicity. Im- proving this could potentially make our current monotone model much more expressive. (2) In Theorem 3.1 the parameter m describes how monotone the model is. Is it possible to use a the model is "boundedly non-monotone", but still enjoys favorable convergence property? (3) Our model currently only learns conditional probability. Is it possible to make it model joint probability efficiently? One way is to mimic PixcICNN: let P(x)) = IT P(xnl=">>) This is inefficient for us in both inference and training, is there way to improve? (4) Although we have a fairly efficient implementation of prox it is still slower than normal nonlinearities like ReLU or softmax. Is there a way to efficiently scale mDBMs? (5) Tsuchida & Ongl (2022) explores the connection between PCA and DEQs, what are other probabilistic models that can also be expressed within the DEQ framework?
to ensure that
(6) Bechmark mDBM image imputations 2019); Richardson al. (2020
together with Yoon et al. (2018): Li et al. (2199): Mattei & Frellsen
12

---

Published in Transactions on Machine Learning Research (04/2023)
5 Conclusion
In this work, we give a monotone parameterization for general Boltzmann machines and connect its mean- field fixed point to a monotone DEQ model. We provide a mean-field update method that is proven to globally convergent. Our parameterization allows for full parallelization of mean-field updates without restricting the potential function to be concave, thus addressing issues with prior approaches. Morcover, we allow complicated and hierarchical structures among the variables and show how to efficiently implement them. For parameter learning, we directly optimize the marginal-based loss over the mean-field variational family, circumventing the intractability of computing the partition function. Our model is evaluated on the MNIST and CIFAR-10 datascts for simultancously prodicting with missing data and imputing the missing data itself. As a demonstration of concept, we also deliver several rllustrations of interesting future directions.
References
David H Ackley, GeofTrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cogmitive science, 9(1):147-11) 169, 1985.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. aie preprint orXis:1909.0137,, 2019.
Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Multiscale deep equilibrium models. arXiv:2006.08633 2020.
arXiv preprint
Pierre Baqué, Timur Bagautdinov, François Fleuret, and Pascal Fua. Principled parallel mean-field inference for discrete random fields. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. -848-5857, 2016.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859 877, 2017.
Robert M Corless, Gaston H Gonnet, David EG Hare, David J Jeffrey, and Donald E Knuth. On the lambertw function. Advances in Computational mathematics, 5(1):329 359, 1996.
Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class- loss based on effec- tive number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9268 9277, 2019.
Justin Domke. Learning graphical model paramcters with approximate marginal inference. IEEE transactions on pallern analysis and machine intelligence, 35(10):2454 2467, 2013.
Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, and Alicia Tsai. Implicit deep learning. SIAM Journal on Mathematics of Data Science, 3(3):930-99) -5880 2021.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Newral computation, 14(8):1771-1800 2002.
Daphne Koller and Nir Friodman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Philipp Krähenbühl and Vladlen Koltun. Parameter learning and convergent inference for dense random fields. In International Conference on Machine Learning, pp. 513 521. PMLR, 2013.
Miguel Lázaro-Gredilla, Wolfgang Lehrach, Nishad Gothoskar, Guangyao Zhou, Antoine Dodicu, and Dileop George. Query training: Learning worse model to infer better marginals in undirected graphical models with hidden variables. orXiv preprint arXiv:2006. X00:20006006003, 2020.
Khuê Lê-Huu and Karteek Alahari. Regularized frank-wolfe for dense erfs: Generalizing mean field and beyond. Advances in Neural Information Processing Systems, 34, 2021.
13

---

Published in Transactions on Machine Learning Research (04/2023)
Steven Cheng-Xian Xian Li, Bo Jiang, and Benjamin Marlin. Misgan: Learning from incomplete data with generative adversarial networks. orXiv preprinl arxiv: 1902. 0902999 2019.
Pierre- Alexande Mattei and Jes Frellsen. Miwae: Deep generative modelling and imputation of incomplete data sets. In International conference on machine learning, pp- 4413 4423.423. PMLR, 2019.
Mohammad Norouzi, Mani Ranjbar, and Greg Mori. Stacks of convolutional restricted boltzmann machines for hhift-invarian feature learning. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.
IEEE, 2009.
Max Revay, Ruigang Wang, and lan R Manchester. Lipschitz bounded equilibrium networks. arXic preprint arXiv:2010.01790 2020.
Trevor W Richardson, Wencheng Wu, Lei Lin, Beilei Xu, and Edgar A Bernal. Mcflow: Monte carlo flow models for data imputation. In Proceedings of the IERE/CVF Conference Computer Vision and Paltern Recognition, pp. 14205-14214 2020.
Ernest K Ryu and Stephen Boyd. Primer on monotone operator methods. Appl. Comput. Math, 15(1):3 43, 2016.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Artificial intelligence and statis- lics, pp. 448 455. PMLR, 2009.
Idan Schwartz, Alexander G Schwing, and Tamir Hazan. High-order attention models for visual question answering. Xiv preprinl arXiv:1711.04329 2017.
Russell Tsuchida and Cheng Soon Ong. Deep equilibrium models estimators for continuous latent variables. arXiv preprint arXtv:2811.0544, 2022.
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukeuoglu. Pixel recurrent neural networks. In Interna- tional Conference on Machine Learning, pp. 1747-1756. PMLR, 2016.
Homer F Walker and Peng Ni. Anderson acceleration for fixed-point iterations. SIAM Journal on Numerical Analysis, 49(4):1715-177) 2011.
Ezra Winston and J Zico Kolter. Monotone operator equilibrium networks. arXie preprint arXiv:2006.0859. 2020.
Zhirong Wu, Dahua Lin, and Xiaoou Tang. Deep markov random field for image modeling. In European Conference on Computer Vision, pp. 295 312. Springer, 2016.
Jinsung Yoon, James Jordon, and Mihacla Schaar. Gain: Missing data imputation using generative adver- sarial nets. In International conference on machine learning, pp. 5689 5698. PMLR, 2018.
Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip HS Torr. Conditional random fields as recurrent neural networks. In Proceedings of the IEFE international conference on computer vision, pp. 1529-1537 2015.
14

---

Published in Transactions on Machine Learning Research (04/2023)
A Appendix
A.1 Deferred proofs
Theorem 3.2. Given f as defined in Equation S a E 10, and x E RR, the prozimal operator prox (2) is given by
(1-a
exp 24 (a==a))))
I-a W where X E R is the unique solution chosen to ensure that the resulling the principal branch of the Lambert W function.
a
a
Ei proxf(a;)= = 1, and where W() is
Proof. By definition, the proximal operator induced by f (the same f in Equation and a solves the following optimization problem:
2 1 - =1|2 + a Σ 2i log 21
a ||=1
min
s.t.
2, 2 0, i ......dd
Σ 2 =1
of which the KKT condition is
xz + 24 + a + aloog 2; - -az + x 4- = for i i [d]
2 0, O,
Σ m = 0, KEld
Σ
=1
i=1
We have that n4 0 is feasible and the first equation of the above KKT condition be massaged as
+ + a + a log 2; 02; + A - 14 = 0
(1 a²2; + a loggoga = =;
x
(1 a)4 log 2;
Σj a a A
[1 2)2; + a log 4
exp
exp
a
Q A
exp
2
= exp
1 2i exp
1 - a
A
exp
0
1 2, W
1 a
a
exp
where W the lambert W function. Notice here 2, 0. Our primal problem convex and Slater's condition holds. Hence, we conclude that
a W
1- exp
E; a - 1 ))
A.2 Convolution network
It is clear that the monotone parameterization in Section directly applies to fully-connected networks, and all the related quantities can be calculated easily. Nonetheless, the real power of the DEQ model comes in
15

---

Published in Transactions on Machine Learning Research (04/2023)
when we more sophisticated linear operators like convolutions. In the context of Boltzmann machines, the convolution operator gives edge potentials beneficial structures. For example, when modeling the joint probability of pixels in an image, it is intuitive that only the nearby pixels depend closely on each other.
Let A E R denote a convolutional tensor with kernel size r and channel size h let x denote some input. For a convolution with stride 1, the block diagonal elements of AT simply form a 1 1 convolution. In particular, we apply the convolutions
(A(x+)) + A(x))
(18)
where À is a x 1 convolution given by
At.:1 = Σ 2 (l',,,,, A,,,,,
(19)
We can normalize by the spectral norm of À term to ensure strong monotonicity. Since À can be rewritten as k x k matrix and k is usually small, its spectral norm can be easily calculated.
It takes more effort to work out convolutions with stride other than 1. Specifically, the block diagonal terms do not form 1 x 1 convolution anymore, instead, the computation varies depending on the location. It is easier to see the computation directly in the accompanying code.
Grouped channels It is crucial to introduce the concept of grouped channels, which allows us to represent multiple categorical variables in a single location, such as the three categorical variables representing the three (binned) color channels of RGB pixel. In this case, each of the three RGB channels will be represented by different group of k channels representing the & bins. The grouping is achieved by having the nonlinearity function (softmax) applied to each group separately. We remark that the convolutions themselves are not grouped, otherwise none of the red pixels would interact with green or blue pixels, etc. Instead, we want all RGB channels to interact with each other (except. that channel i at position (j, k) does not interact with itself). That means in Equation (3), the blkdiag( A" Á) is grouped in the following way. Recall that this block diagonal term has element size k; x k; for E [m]. This parameterization has only 1 group. With g groups, the element of the block diagonal matrix then has size k; .i....., k,, , x ki for i e [n], where Ejerol Ki, kg We also observe empirically that grouping the latent variables improves the performance.
A.3 Efficient computation of prox
The solution the proximal operator in damped forward iteration given in Theorem 3.2 involves the Lambert W function, which does not attain an analytical solution. In this section, we show how to efficiently calculate the nonlinearity (II), as well as its Jacobian matrix for backward iteration.
Let f(y) = - W (2=0 exp (% - 1)), and we have
x
log f(y
a log 1 I-a
+ log 2/aa1 log
1 1-c a W (1- exp
( -11 1)
y
where the last equality uses the identily log(W(x)) = log bogx-Wor (x)). Rewrite W (L exp (3 - 1)) = f(v) 1a and massage the terms, we have that solving log f(y) is equivalent to finding the root of
Mxx) = y a e" a) - ax.
Direct calculation shows that (1 (1 a) and h² - a)e"" Note here y is the input and it is known to us, and is scalar. Hence we can efficiently solve the root finding problem using Halley's method. For backpropagation, we need di dy
which can be computed by implicit diffcrentiation:
h(z)= y a ea(1 a) ax =
dr 1
dy at (1 a)ez y ax
16

---

Published in Transactions on Machine Learning Research (04/2023)
= Now can find À s.t =1 using Newton's method on g(A) = E, clol(((osst)) -1= 0. Note this is still a one-dimensional optimization problem. A direct calculation shows that dy dd = E, clo ((
dA
and above we have already calculated that
d dlog(s(a + 2)
dx
1
dA
dy
y + 2 ax
For backward computation, by the chain rule, have:
lo f(a++A)
d'log(f(e + A))
dx;
dæ
1 dA/dr;
eog ((x+t)
24
+ a log(f A))
where the last step derived by implicit differentiation. Now to get dd/dai, notice that by applying the implicit function theorem p(æ, X(x)) = E, loB(f(a; - O, we get.
dA
dp
dp
=
dx;
dA
dr:
Thus have all the terms computed, which finishes the derivation.
B Additional Experiments and Details
Here we provide the model architectures and experiment details omitted in the main text.
B.1 Details
Model architectures For MNIST experiments (except for the extrapolation), using the notation Equa- tion the mDBM consists of 4-layer deep monotone DEQ with the following structure:
A
0
0
A21
Ag2
0
0
A31
Ag2
A33
0
0
0
A43
A44
where An is a x 20 3 x convolution, A22 a 40 x x 3 convolution, A21 40 20 x 3 convolution, A3ı is 80 20
convolution with stride 2, Aa3 is 80 80 stride 4, Ayz is 80 x 40 3 x 3 convolution with stride 2, Aus is (80- 7- 7) 10 dense linear layer, and A44 is 10 10 dense linear layer. The corresponding variable q as in Equation g then has 4 elements of shape (20 x 28 x 28), (40 x 14 x 14), (80 x 7 x 7), (10 x 1). When applying the proximal operator to q, we use 1, 10, 20, as their number of groups, respectively.
convolution with
The deep RBM consists of 3-layers where the first hidden layer has 300 neurons, and the last hidden layer (representing the digits) has 10 neurons, amounting to in total 239,294 paramcters. For comparison, the mDBM has 192,650 parameters.
The mDBM used CIFAR-10 is the same as for the MNIST experiments with the following exceptions: An is 20 x 20 3 3 convolution, Az is a 24 x 24 3 3 convolution, Az is a 24 x 20 x x convolution with stride 2, A33 a 48 x 48 x 3 x 3 convolution, Aaı is 48 x 20 x 3 x 3 convolution with stride 4, A is a 48 x 24 x 3 x 3 convolution with stride 2, A43 a (48 8 8.S. 8) x 10 dense linear layer, and A4 is a x 10 dense linear layer. The corresponding variable q as Equation 9 then has elements of shape 8), (10 1). When applying the proximal operator to q, we use 1,6, 12, their number of groups, respectively.
(60 32 x 32), (24 16 x 16), (48
17

---

Published in Transactions on Machine Learning Research (04/2023)
For the MNIST extrapolation experiments, use mDBM of the following structure:
A11
0
0
A21
A22
0
0
A31
Ag2
33
0
A41
A42
A43
A44
where An is 4 x 3 x 3 convolution, A22 is a 40 x 40 x convolution, A21 is a 40 x x 3 x 3 convolution with stride 2, Ags is 80 x 80 3 x 3 convolution, Ay1 is 80 x 4 x 3 convolution with stride 4, A32 is 80 x 40 x 3 x 3 convolution with stride 2, A41 is (4. 28 28) x 100 dense linear layer, A42 is a (40 - 14- 14) x 100 dense linear layer As is a (80 7- x 100 dense linear layer, and A4s is a 100 x 100 dense lincar layer. The corresponding variable q as in Equation ( then has 4 elements of shape
(4 x 28 x 28), (40 x 14 x 14), (80 x 7 x 7). (100 x 1). When applying the proximal operator to q, we use
1, 10 20, 10 as their number of groups, respectively.
The model used for CIFAR-10 extrapolation has the same strueture, where Au is 30 30x30x3x3 convoluttio, A22 is a 60 x 60 x 3 x 3 convolution, A21 is 60 x 30 x 3 x 3 convolution with stride 2, Aas is 80 x 80 x 3 x convolution, A31 is a 80 x 30 x 3 x 3 convolution with stride 4, A32 is 80 x 60 x 3 x 3 convolution with stride 2, A41 is a 220-32-12 -22-30 32) x 100 dense linear layer, A42 is a (60- (60-16- 16 16) x 100 dense linear layer A43 is a (80-S 8 8) x 100 dense linear layer, and Ass is a 100 x 100 dense linear layer. The corresponding variable q as in Equation (9) then has 4 elements of shape (30 x 32 x 32), (60 x 16 x 16), (80 x 8 x 8), (100 x 10). When applying the proximal operator to q, we use 1,6, 8, 10 as their number of groups, respectively.
mDBM Training details and hyperparamcters Treating the image reconstruction as a donse classifi- cation lask, we use cross-entropy loss and class weights T-5"r with / = 0.9999 (Cui et al. 2019), where n; is the number of times pixels with intensity i appear the hidden pixels. For classification, we use standard cross-entropy loss. To enable joint training, we put equal weight of 0.5 on both task losses and backpropa- gate through their sum. For both tasks, we put T. into the cross-entropy loss as logits, described in Equation 17 Since mean-field approximation is (conditionally) unimodal, the scaling grants us the ability to model more extreme distributions. To achieve faster damped forward-backward iteration, we implement Anderson acceleration (Walker & Ni. 2011), and stop the fixed point update as soon as the relative difference between two iterations (that is, |9:+1 - q: l/1l/lll is less than 0.01, unless we hit a maximum number 50 allowed iterations. For proxž and the damped iteration, we set = 0.125 (Although one can tune down a whenever the iterations do not converge, empirically this never happens on our task).
We use the Adam optimizer with learning rate 0.001. For MNIST, train for 40 epochs. For CIFAR-10. we train for 100 epochs using standard data augmentation; during the first 10 epochs, the weight on the reconstruction loss is ramped up from 0.0 to 0.5 and the weight on the classification loss ramped down from 1.0 to 0.5; also during the first 20 epochs, the percentage of observation pixels is ramped down from 100% to 50%.
Deep RBM Training details and hyperparameters The deep RBM is trained using CD-1 algorithm for 100 epochs with a batch size of 128 and learning rate of 0.01.
Convergence of inference during training We note that, comparing to the parameere--p monDEQ in Winston & Kolter (2020), whose linear module suffers from drastically increasing condition number (hence in later epochs taking around 20 steps to converge, even with tuned o), our parameterization produces a much nicer convergence pattern: the average number of forward iterations over the 40 training epochs is less than 6 steps, see Figure 7
mDBM patch imputation experiments We train mDBM on the task of MNIST patch imputation. We randomly mask a 14 x 14 patch, chosen differently for every image, similar to the query training in et al. (2020). To make the model class richer, we lift the monotonicity constraint, and find that the model converges regardless. Our model reconstructs readable digits despite potentially large chunk of missing pixels (Figure Sb). If the model is given the image labels as input injections, our model
18

---

Published in Transactions on Machine Learning Research (04/2023)
Backward iterations
iberations.
25
farations
1
15
0
2500
5000
7500
10000
12500
15000
17500
Figure 7: Convergence of forward-backward splitting.
7
2
4
4
1
4
3
7
2
6
4
4
9
7
J
f
u
3
5
7
0
6
9
0
1
5
5
9
0
6
9
0
5
17
L
7 7
5
0
7
3
4
7
5
4
5
9
7
8
4
9
6
4
5
tr
4
0
7
9
0
1
4
0
7
4
0
34733
3
4
7 2 7
7
Z
3
4
7 2 7
2
(a) Test data with 14 x 14 patch masked
(b) Imputation with a 14x 14 patch masked, inference without injection labels
(c) Imputation with a 14x 14 patch
masked, inference with injection la- bels
Figure 8: MNIST pixel patch imputation using mDBM
performs conditionaly generation fairly well (Figure Sc These results demonstrate the flexibility of parameterization for modelling different conditional distributions.
Deep RBM results using Gibbs sampling The deep RBM is trained as before. For joint imputation and classification, the DBM uses Gibbs sampling of 10000 and 100 steps respectively, although the quality of the imputed image and test accuracy are insensitive the number steps.
We randomly mask off 60% pixels, or randomly selected 14 14 patch; the results shown in Figure and are better than when mean-field inference is used, (shown in Figure 3
In the experiment with 60% of pixels randomly masked, we also test the model on predicting the actual digit simultaneously. The test accuracy is 93.58%, comparable to the mDBM accuracy of 92.95%.
Comparison of inference methods We conduct numerical experiments to compare our inference up-
dating method to the oncs proposed by Krähenbühl & Koltun (2013); Baqué et al. (2016). denoted Krähenbühh's and Baqué's respectively. Krähenbühl's fast concave-conves procedure (CCCP) essentially decomposes to Equation (1)), the un-damped mean-field update with softmax. This update only converges provably when & is concave. Baqué's inference method can be written as
(t) h
softmax (1 log di (t-1) a a(#nng)
((-1) + P + bn))
(20)
This algorithm provably converges despite the property of the pairwise kernel function. However, this procedure converges in the sense that the variational free energy keeps decreasing, Therefore their fixed point may not be the true mean-field distribution Equation (7). In this experiment, we train the models using three different updating methods, and perform inference using three methods well, with and without the monotonicity condition.
19

---

Published in Transactions on Machine Learning Research (04/2023)
7
2
/
0
4
1
Y
9
7
2
1
6
4
1
4
9
4
25
25
25
6
9
0
6
9
0
1
5
5
9
0
6
9
0
1
5
0
50
75
9
7
3
4
9
6
5
75
9
7
8
4
9
6
6
5
75
4
0
7
4
0
3
4
0
7
4
0
3
125
3
7
2
7
2
3
3
4
2
7
2
1
2
150
150
150
50
100
150
200
50
100
150
50
100
150
(a) 60% pixels are randomll masked. From left to right:
imputeed image, true image, masked image.
7
2
1
6
4
1 4
9
7
2
1
0
4
1
4
9
25
50
5
9
0
6
0
1
5
50
5
9
0
6
9
0
1
5
50
2
J
75
9
7
9
4
9
6
4
5
75
9
7
8
4
9
6
6
5
75
C
o
4
0
7
4
0
3
4
0
7
4
0
3
1
125
3
4
7
2
7
2
125
3
4
2
2
l
125
150
100
150
100
150
(b) 14 14 patches are randomly masked. From left right: imputed image, true image, masked image.
Figure 9: RBM for image imputation using Gibbs sampling
We also compare the convergence our method to the regularized Frank- Wolfe method Lê-Huu Alahari h021b. Their update step can be written as
(++1) q
(1 og + softmax (omg))) (1) + Ino + by))
(
We use À = 0.7 as in their paper. Our method converges faster than the FW method. See the result in
Figure 10
Our Regularized FW
Number ittrations
10
0.10
0.08
0.06
0.04
0.02
0.00
'lleancc (decreasing)
Figure 10: Convergence of our method vs. the regularized FW. This experiment is done using the same setup as in Figure II
Krähenbühl's and Baqué's methods often do not converge in the backward pass (there's no theoretical guarantees neither). To rule out the impact of the backward iteration, during training we directly update use the gradient of the forward pass, instcad of using backward gradient hook to compute Equation (16). [2gure[2aan and Figure demonstrate how the three update methods impute missing pixels when trained with different update rules, with and without the monotonieity condition, respectively. Krähenbühl's usually does
20

---

Published in Transactions on Machine Learning Research (04/2023)
Table 4: Relative update residual when monotonicity is not enforced
Inference
Krähenbühl
Baqué
Our
Train
Krähenbühl
0.0005
0.0065
0.0024
Baqué
1.0924
0.0119
0.0042
mDBM
1.1286
0.0065
0.0022
Our
50
0.010
Baque
Number ot iteratiins
40
0.008
30
TV
0.004
20
0.002
10
0.000
0.10
0.08
0.06
0.04
0.02
0.00
0.10
0.08
0.06
0.04
0.02
0.00
Telerance (decreasing)
Telerance (decreasing)
(a) TV distance when trained
(b) Number of iteration till convergence when trained
0.085
35
Baque
Number of iterations
0.070
25
0.065
20
15
0.055
10
0.050
0.10
0.08
0.06
0.04
0.02
0.00
0.10
0.08
0.06
0.04
0.02
0.00
(decreasing)
(decreasing)
(c) TV distance at initialization
(d) Number of iteration till convergence at initialization
Figure 11: TV distance and convergence speed
not converge when the model is trained with our method Baqué's, s, whereas the other two methods impute the missing pixels well. The classification results are presented in Table and Table i Notice that when trained with our method or Baqué's, the convergence issue of Krähenbühl's 's leads to horrible classification accuracy. Our method is superior to other inference methods when the model is trained in a different update fashion. For example, if the model is trained by using Krähenbühl's it makes sense that the model performs the best if the inference also Krähenbühl's since the parameters are biased toward that particular inference method. However, our method in this case outperforms Baqué's.
After these methods halt and return of we one more iteration of
+1 softmax (1 + Pho + bn),
(21)
and record the relative update residual || T+1 of ll/l/ for randomly selected 4000 MNIST images. The results are listed in Table 3 and Table | To alleviate the effect of numerical issues, we strength the convergence condition either the relative residual is less than 1 or the number of iterations exceeds 100 steps.
21

---

Published in Transactions on Machine Learning Research (04/2023)
7
Z
0
4
4 9
7 2
1
0
4
4 & 8 4 3 1 / 0
1
7 2
/
6
4
1
4
&
Krähenbill
9
0
6
9
0
5
5 9
0
6
9
0 6
5
$ 9
0
6
9
0
5.
97
8
4 4
4
6 1
4 3 2 4 1 8 3
5
9
7
8
4
4 0
5
9 7
8
8
4
6
4 3 2 4
5 1 I a
4
D
7
0
4 107
7
4
1
4 0
4
0
1
3 L/ 7
2
7
3 4
7
2
1
3
4
7
7 ~
4
4
/ 0
7
2
/
0
4
4 1 6 3 2 2 7 1
9 5
7
2
/
0
4
/
$ 2 0
8
7
8
6
9
0
6
9
9
0
6
9
0
5 5 1
Baqué
1 7
9 7
8
4
9
6
5
9 7
8 07
4
7
6
6
8
3 1 2
4 0
4
0
3
0
0
1
3 2 4
3
2
7
2
3 4
1
7 7
7
0
4
/ 0
9
7
2
0
4
9 5
6
9
0
6
15
9 7
0
6
9
0
Our
5
9
7
8
4
9
6
4
5
9
8
4
9
6
4
5
4 0
7
4
0
3
1
4
0
7
4
0
3
1
3 4
7
2
7
1
3
3 4
7
2
7
1
3
1
(a) Krähenbühl
(b) Baqué
(c)
Our
Figure 12: Training and inference using all three update rules with 40% observed pixels with the monotonicity
condition. The labels on cach row represent the training update rule, and the labels on the columns represent the inference update rule.
Table 5: Classification error (standard deviation) when monotonicily is enforced
Inference
Krähenbühl
Baqué
Our
Train
Krähenbühl
0.042 (0.0013)
0.114 (0.0019)
0.0498 (0.0014)
Baqué
0.958 (0.0013)
0.038 (0.0010)
0.034 (0.0012)
mDBM
0.946 (0.0024)
0.0425 (0.0016)
0.0412 (0.0017)
It appears in Table 3 and Table 4 that although our method has a much lower residual compare to Baqué's, both them seem small and convergent. This because the "optimal" fixed point in this setting on MNIST might be unique and both methods happen to converge to the same point.. However, this is in general not truc. We compare our method Baqué's on 400 randomly selected MNIST test images with 40% pixels observed, and perform mean-field update until the relative residual of [0.1, 0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001] is reached (without step constraint), respectively. Then we measure the TV distance between the distributions computed by these two methods on the remaining 60% pixels, as well the convergence speed. The results are demonstrated in Figure 11 One can see that when the model trained (using Krähenbühl's, Figure 11a the TV distance converges to as the tolerance decreases. However, when the model just initialized (but still constrained be monotone), the TV distance remains large (Figure 11ch. Even though in this case the optimal fixed point may be unique, our method is still superior Baqué's: it takes less iterations till convergence, despite whether the model is trained not.
22

---

Published in Transactions on Machine Learning Research (04/2023)
7
7
6
4
4 9
7 2
1
6
4
4
7 2
1
6
4
4
Krähenbüll
9
0
6
9
0
}
3
5 9
0
6
4
0 6
1 10 3 2 4 A
3
4 9
0
6
4
0
33
97
8
4
9
1
5
9 7
8 7
4 4
9 0
5
9 97.9
8
4
9
6
10
5 1 / A
4
0
7
4
0
3
4 0
4 000
4
0
3 2 4
3 4
7
2
7
2 1
3 4
7
2 0 6 1
|- 1
3 4
1
2
7 2
/ 0
4
7 2
/
0
4
/
9
0
6
8 01
5 3
5 9
9
0
1
5
$ 9
0
6
9
0
5 5 1
Baqué
9
8
7
6
38 37
9 7
8
9
6
4
5
9 7
8
1
9
6
4
86
4 4
3
0
3
1
4 0
4004
0
3
3 L
4
34727
4
|
1
7
2
/
0
4
1
4 1
9
7
2
/
0
4
/
4
a
5
9
0
6
9
0
5
5
9
0
6
9
0
5
9 7
3
4
1
6
00
5
4 7
3
1
6
Co
5 1
4
0
7
4
0
3
4 0
7
4
0
3
3
4
7
2
7
2
1
3 4
7
2
7
2
1
(a) Krähenbühl
(b) Baqué
(c) Our
Figure 13: Training and inference using all three update rules with 40% observed pixels without the mono- tonicity condition. The labels on each row represent the training update rule, and the labels on the columns represent the inference update rule.
23

---

Published in Transactions on Machine Learning Research (04/2023)
Table 6: Classification error (standard deviation) when monotonicity is not enforced
Inference
Krähenbühl
Baqué
Our
Train
Krähenbühl
0.035 (0.0017)
0.189 (0.0023)
0.051 (0.0015)
Baqué
0.762 (0.0013)
0.041 (0.0013)
0.055 (0.0012)
mDBM
0.90 (0.0002)
0.063 (0.0021)
0.036 (0.0017)
24


Reviews:
[{"review_id": "jOa5OishLR", "reviewer": "Reviewer_zuLa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed method seems to be technically sound and is well explained mathematically. It also draws from a much newer, powerful class of DEQ models that seem to work well in practice.\n\n- I like the idea of constructing a highly structured parameterization to ensure that the necessary conditions are satisfied immediately.\n\nWeaknesses:\n\n- The experiments seem to be quite limited, especially compared to the current powerful methods of diffusion models in generative modelling and even DEQ models, which can be used for sequence modelling as well. I guess I'm asking what can a (m)DBM do that these approaches cannot?\n\n- There's no clear experiments on convolutional nets (I could be missing them?) although that is listed as a selling point of the approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Sorry for the late review here.\n\nThis paper proposes monotone deep boltzmann machines (mDBMs), which are a restriction of deep Boltzmann machines to the class of graphs that have no self-loops within a layer, but can connect both forwards and backwards in the layers. mDBMs have the favorable property that they can be parameterized as a deep equilibrium model (DEQ), which gives reasonably efficient training procedures and global solutions. \n\nExperiments are performed in MNIST and CIFAR-10 in an unsupervised fashion, demonstrating that this inference procedure is quite flexible and reasonably accurate.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested experiments\n\n- My understanding is that DEQ based models (e.g. https://arxiv.org/pdf/2006.08591.pdf)  and some of the DBM results (e.g. https://www.scinapse.io/papers/2100495367) tend to have experiments in classification or regression using the DBM as a feature constructor. If this is possible with the mDBM, could this be done here? That is, classifying cifar-10 classes based on the states from the mDBM, not just in-filling.\n\n- Figure 4: what is the extrapolation performance of the mDBM on MNIST and CIfAR-10? For example, mask the left 50% of pixels and then attempt to generate the right half.\n\nThese are mostly presentation based comments:\n\n- pg 4: redefine phi, I believe it's the connection network of the graph?\n\n- pg 5: \"We see that q_h is the ...\": explain this further as it seems both important and not obvious at first glance\n\n- pg 5: where is the proof of proposition 3.1? mark this in the main text\n\n- eq 8: use \\mathbb{I} instead of I\n\n- pg 6: \"height x ... x \" use $x$ for the x-es here\n\n- pg 7: how are m and L determined prior to solving the system? I think m might be mentioned above, but L is not\n\n- pg 7: give a sentence describing Halley's method\n\n- pg 7: \"extend their work to the softmax nonlinear operator\": I thought that your work used the proximal operator instead of the softmax one?\n\n- pg 8: \"This backward pass can also be computed via an iterative approach...\" explain what their approach is\n\n- tables 1,2: Why do the errors go down as the masking percentage goes up? \n\n- tables 1,2: please replace \"our\" with mDBM\n\n- table 3: What does relative residual update mean in this context? I'm also more generally confused by the \"comparison of inference methods\" section. Is the conclusion that your method and Baque's methods are much more globally accurate?\n\n- pg 11: \"although we have a fairly efficient implementation ...\" why not one step newton approach or construct a polynomial/ taylor series  approximation? It seems like there are well-known approximation methods for the Lambert W function, like [this one](https://nvlpubs.nist.gov/nistpubs/jres/65B/jresv65Bn4p245_A1b.pdf). ", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q6IuEAiKx1", "reviewer": "Reviewer_djWs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: \n- The paper is overall well written paper, relatively easy to follow. \n- Elegant formulation and nice application of recent advances on deep learning methods to more classic models, like DBMs\n- Some implementation details needed to apply the results in Winston & Kolter (2020) are carefully explained, providing from a methodological point of view a nice contribution\n\n**Weaknesses**\n- Limited experimental evaluation (see below)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The Authors present a new approach for deep Boltzmann machines, which leverages recent advances on equilibrium models to propose a novel inference scheme for DBMs. This is achieved by applying the results of Baie et al. (2019) and Winston & Kolter (2020) regarding fixed-point convergence of infinitely deep neural networks with input injection (of the class $\\sigma(Wz+Ux+b)$) to DBMs, allowing to derive a scalable variational inference scheme. This involves constraining the linear model to be compatible with the DBM architecture (i.e. graphical model with no-loops) and to ensure the requirement for Winston & Kolter (2020) to hold (i.e. monotonicity). After an in-depth explanation of the proposed method, the Authors conclude the paper with a small section with experiments on MNIST and CIFAR datasets, mostly on image imputation and marginally on image classification.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Evidence of convergence to fixed point: beyond the theoretical guarantees, would it be possible to show the fixed point convergence empirically?\n- Effect of temperature: you quickly mentioned some temperature scaling to the output marginals but you didn't, as far as the paper goes, tested different configurations. Didn't you find some pathological situations where this was necessary? What would happen with different setups?\n- Summary of the method and algorithm: the developed method involves several steps and different parameterizations and approximations; it would be nice if you could provide a paragraph at the end of section 3 to summarize the proposed method, maybe with the aid of a pseudo-algorithms.\n- Additional experimental evaluation: the paper for the moment has a limited experimental campaign, based only on 2 datasets and 1 comparison (the naïve RBM). One possibility is to benchmark mDBM with other generative models for e.g. image imputation (see 1-4 for some ideas). To be clear, for me the paper has value beyond the ranking on this benchmark, but I would encourage the Authors to explore with different models. This would be also a good opportunity to briefly discuss and compare computational aspects of your proposal.\n\n\n(1) GAIN: Missing data imputation using generative adversarial nets. ICML 2018\n\n(2) MisGAN: learning from incomplete data with generative adversarial networks. ICLR 2019\n\n(3) MIWAE: Deep generative modelling and imputation of incomplete data sets. ICML 2019\n\n(4) MCFlow: Monte Carlo Flow Models for Data Imputation. CVPR 2020", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kHCVUXZm8C", "reviewer": "Reviewer_ZkY7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The question of whether more general Boltzmann Machines than RBMs admit efficient inference procedures is an interesting one. This paper answers that question in the positive, although perhaps in a somewhat restricted setting. It leverages other parts of the literature, namely Deep Equilibrium Models, in order to do so, which I think is nice.\n- The paper is mostly well written. Excluding isolated instances, it is easy to follow the train of thought and notations of the authors.\n\n**Weaknesses:**\n- Section 3.3 does not clearly describe the conversion of ``neural network convolution operators'' to parameterisation in terms of $A$. I do appreciate that this is a difficult thing to convey, due to the various subscripts involved. Also the mention of the spectral normalisation of the neural net style convolution operator. I encourage the authors to spend some time thinking about the best possible way to communicate this, because if this is done well, it will be very useful for other researchers. Deferring the explanation to the Appendix is suboptimal.\n- As the authors admit in the abstract, the work is largely at a conceptual stage. The datasets and problems considered are largely toy from a ``deep learning perspective'', but not from a `graphical model' perspective. I think it is good that the authors acknowledged this weakness in the abstract, experimental evaluation, and somewhere else towards the beginning, and I do not see this as a substantial enough weakness to prevent publication at TMLR.\n\n**Questions** (for clarification --- not necessarily strengths or weaknesses):\n- First paragraph. Are DBMs necessarily discrete? I seem to recall some versions allow for continuous-valued nodes.\n- Figure 1, right. Why did you choose to draw a general BM in this way? It appears as though what you have drawn is not a complete graph, unless I misunderstand the notation or terminology. For example, the top right node is not connected to the top middle node. I am more familiar with the layout where the nodes are arranged on a circle, and every node is connected to every other node.\n- Equation (4). What is $\\Phi_{ij}$? The matrix $A_i$ is a $d \\times k_i$ matrix, so I guess $\\Phi_{ij}$ should be a $k_i \\times k_j$ matrix? So then the ``0'' listed in the second line of equation (4) is actually $0_{k_i \\times k_j}$? By the way, I think there is a minor typo in the text two lines after equation (1). You write $\\Phi_{i,j}$ but I think it should be $\\Phi_{ij}$.\n- The monotone DEQ construction is guaranteed to admit a unique fixed point under a certain condition like $I - \\Phi \\succ m I$, which translates to a certain condition on $A$ or $A$ hat. I believe these conditions are easy to impose by always enforcing the spectral constraint and then running any gradient based optimiser. Is that true? Or does the gradient based optimiser have to explicitly enforce a condition on the parameters it is learning?\n- Perhaps some of the works below and the others that you can find might be useful for your aspiration (4) in Future directions. Namely, other activation functions can be obtained using other frameworks that still allow for the existence of a unique fixed point. Some of these also admit probabilistic interpretations.\n\nAll in all, I am leaning to accept this paper. I think some of the audience of TMLR will find it very interesting, and I find the claims to be without any serious issues.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Boltzmann machines are a classical fully-connected model in which computing (properties of, or sampling from) stationary distributions is intractable. \nRestricted Boltzmann machines and their deep counterparts wherein connections are organised in a layered structure allow for tractable inference via a form of Gibb's sampling called contrastive divergence. \nThe authors consider another type of connectivity structure that allows for tractable inference.\nThey call the structure the monotone DBM, where arbitrary self-connections are allowed in each layer but the weights are restricted in such a way as to allow a unique fixed point.\nThis is achieved through a connection to recently proposed implicit layers called deep equilibrium models, and in particular, monotone deep equilibrium models with special activation functions.\nMonotone DBMs are applied to joint image completion and classification tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Minor recommendations:**\n- Start of section 2. \"Deep equilibrium models, especially their convergent version, the monotone\". The phrase \"their convergent version\" seems to imply there is a singular convergent version, but there exist multiple. For example, [1-4] and others that are recently published.\n- Theorem 3. The first part of the theorem 1) $\\Phi_{ii} = 0$ (which I believe should be a matrix of zeros) is trivial, by (4). Why include it in the theorem statement?\n- First equation in section 3.2. Is $E$ the set of all edges? Is $o$ the set of observed nodes?\n- I find the notation where the fixed point of a DEQ $z(x)$ is an explicit function of $x$, but then later in for example equation (15) it is not. I suggest picking one notation and keeping it consistent.\n- Text above equation (17). ``owning'' should be ``owing''.\n- Figure 3. Why are (a) and (d) different figures? It seems like a different seed of a random mask was applied to each. And I believe (c) and (f) are the same. Can you make this figure where there is a single figure instead of (a) and (d), and a single figure instead of (c) and (f)?\n- Why did you compare with mean-field updates on the RBM, if Gibbs sampling/contrastive divergence is the de facto standard? Perhaps move the result from the appendix to the main text?\n\n\n[1] Ezra Winston and J Zico Kolter. Monotone operator equilibrium networks. Advances in Neural Information Processing Systems, 33:10718–10728, 2020.\n[2] Max Revay, Ruigang Wang, and Ian R Manchester. Lipschitz bounded equilibrium networks. arXiv preprint arXiv:2010.01732, 2020.\n[3] Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, and Alicia Tsai. Implicit deep learning. SIAM Journal on Mathematics of Data Science, 3(3):930–958, 2021.\n[4] Tsuchida, R., & Ong, C. S. (2022). Deep equilibrium models as estimators for continuous latent variables. arXiv preprint arXiv:2211.05943.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "e0SZRUvDhp", "reviewer": "Reviewer_iVQk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and presents the ideas and novelties (and shortcomings, connections to related work) in a transparent manner.  Although I did not check the theoretical claims thoroughly, in that respect the paper is quite dense and requires quite a lot of background knowledge, the presentation of the results is concise and seems correct.\nArguable the theoretical novelty is limited but I still like the new connection to DBMs, and the novel theoretical results that allow for this bridge. I think that the paper is well executed and will be well received and appreciated. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors take inspiration of recent work on DEQs allowing for provable convergence and extend these ideas to deep Boltzmann machines. They provide a restricted set of deep Boltzmann machines that crucially allows for recurrent connections. They then provide theory allowing for provable convergence of their mean-field updates and practical insights into how to enable fast inference with their model. The theory is accompanied with classic experiments on vision datasets (MNIST, CIFAR10).   ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have no requested changes although a couple of suggestions. \nIt would be nice to get a bit better feeling of the algorithm and how it compares to related work:\n\nHow fast/slow is the algorithm (and the compared methods) in wall-clock time? It would be nice how long the different components scale with network / dataset size. It would be useful to provide pseudocode for training and inference to allow for an overview of the different algorithmic components. \nIn general, the paper is very much concerned about provable convergence. As mentioned by the authors (and in related work), quite surprisingly at least for me,  the dynamical systems described by the DEQ / your updates seem often to converge without additional regularisation or more elaborate intervention. Could you maybe elaborate on that? I think this part falls a bit short in the paper.\n", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is generally a strong paper with some interesting ideas that I believe is highly suitable for publication at TMLR.  Though reviewers originally raised some concerns about experimental comparisons, these were all things that were either adequately addressed in the rebuttal or that the reviewers did not feel should prohibit acceptance.  As the authors have already made updates to the paper based on the reviewers' comments and there are no significant issues still outstanding, I recommend accepting the paper "as is".
Decision: Accept (poster)