Quantitative Universal Approximation Bounds for Deep Belief Networks
Authors: Julian Sieber
Johann Gehringer
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep belief network
restricted Boltzmann machine
universal approximation property
expressivity
Kullback-Leibler approximation
probability density

Abstract: We show that deep belief networks with binary hidden units can approximate any multivariate probability density under very mild integrability requirements on the parental density of the visible nodes. The approximation is measured in the $L^q$-norm for $q\in[1,\infty]$ ($q=\infty$ corresponding to the supremum norm) and in Kullback-Leibler divergence. Furthermore, we establish sharp quantitative bounds on the approximation error in terms of the number of hidden units.

Under review as a conference paper at ICLR 2023
QUANTITATIVE UNIVERSAL BOUNDS FOR DEEP BELIEF NETWORKS
APPROXIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
We show that deep belief networks with binary hidden units can approximate any multivariate probability density under very mild integrability requirements on the parental density of the visible nodes. The approximation is measured in the -orm for q E [1, oo] (q = oo corresponding to the supremum norm) and Kullback-Leible divergence. Furthermore, we establish sharp quantitative bounds on the approximation error in terms of the number of hidden units.
INTRODUCTION
Deep belief networks (DBNs) are a class of generative probabilistic models obtained by stacking sev- eral restricted Boltzmann machines (RBMs, Smolensky (1986)). For brief introduction to RBMs and DBNs we refer the reader to the survey articles Fischer & Igel (2012; 2014), 4); Montúfar (2016); Hinton al. (2006); Hinton & Salakhutdinov
Ghojogh et al. (2021). Since their introduction, (2006), DBNs have been successfully applied to variety of problems in the domains of natural language processing Hinton (2009); Jiang et al. (2018), bioinformatics Wang & Zeng (2013); Liang et al. (2014); Cao et al. (2016); Luo et al. (2019), financial markets Shen et al. (2015) and com- puter vision Abdel-Zaher & Eldeib (2016); Kamada Ichimura (2016; 2019); Huang et (2019). However, our theoretical understanding of the class of continuous probability distributions, which can approximated by them, limited. The ability to approximate a broad class of probability ussulll referred to as universal approximation property- still an open problem for DBNs with real-valued visible units. As a measure of proximity between two real-valued probability density functions, typically considers the Ld-distance the Kullback-Leible divergence.
Contributions. In this article we study the approximation properties of deep belief networks for multivariate continuous probability distributions which have a density with respect to the Lebesgue measure. We show that, as DBNs with two hidden layers of sizes m and m + 1, respectively. Furthermore, we provide an explicit quantitative bound on the approximation error in terms of m. More specifically, the main contributions this article are:
the universal approximation property holds for binary-
For each E [1, oo) we show that DBNs with two binary hidden layers and parental density ip : IRd 14 can approximate any probability density f Id + R4, in the norno solely under the condition that f. sp E T (RR)), where
- F:: Rd R: |1 ((Rd)) = {
- do) < }
addition, we prove that the error admits a bound order O (mmmmaas -1) for each q (1, oo), where m is the number of hidden neurons.
If the target density uniformly continuous and the parental density y is bounded, provide an approximation result in the -nom (also known as supremum or uniform norm), where
{ Rdd + +R |ff
sup [((=) co).
L 00 (Rd)
zERd

---

Under review as a conference paper at ICLR 2023
Finally, we show that continuous target densities supported compact subset of R and uniformly bounded away from zero can be approximated by deep belief networks with bounded parental density in Bellback--eibler divergence. The approximation error in this case is of order (mm-))
Related works. One of the first approximation results for deep belief networks is due to Sutskever & Hinton (2008) and states that any probability distribution on {0, 1]d can be learnt by DBN with 3 x 2d hidden layers of size d + 1 each. This result was improved by Le Roux & Bengio (2010); 2d-1 Montúfar Ay (2011) by reducing the number of layers to a=log(d) with d hidden units each. These results, however, are limited to discrete probability distributions. Since most applications involve continuous probability distributions, Krause et al. (2013) considered Gaussian- -inary DBNs and analyzed their approximation capabilities in Kullback-Leibler divergence, albeit without a rate. In addition, they only allow for target densities that can be written as an infinite mixture of set of probability densities satisfying certain conditions, which appear to be hard to check in practice.
Similar questions have been studied for a variety of neural network architectures: The famous sults of Cybenko (1989); Hornik et al. (1989) state that deterministic multi-layer feed-forward net- works are universal approximators for large class of Borel measurable functions, provided that they have at least one sufficiently large hidden layer. See also the articles Leshno et al. (1993); Chen & Chen (1995); Barron (1993); Burger & Neubauer (2001). Le Roux & Bengio (2008) proved the universal approximation property for RBMs and discrete target distributions. Montúfar & Mor- ton (2015) established the universal approximation property for discrete restricted Boltzmann ma- chines. Montúfar (2014) showed the universal approximation property for deep narrow Boltzmann machines. Montúfar (2015) showed that Markov kernels can be approximated by shallow stochastic feed-forward networks with exponentially many hidden units. Bengio & Delalleau (2011); Pas- canu et al. (2014) studied the approximation properties of so-called deep architectures. Merkh & Montúfar (2019) investigated the approximation properties of stochastic feed-forward networks.
The recent work Johnson (2018) nicely complements the aforementioned results by obtaining an illustrative negative result: Deep narrow networks with hidden layer width at most equal to the input dimension do not posses the universal approximation property.
Since methodology involves an approximation by convex combination of probability densities, we refer the reader to the related works of Nguyen & McLachlan (2019); Nguyen et al. (2020) and the references therein for an overview of the wide range of universal approximation results in the context of mixture models. See also Everitt & Hand (1981); Titterington et al. (1985); McLachlan & Basford (1988); McLachlan & Peel (2000); Robert & Mengersen (201 I); Celeux (2019) for in-depth treatments of mixture models.
The recent articles Bailey Telgarsky (2018); Perekrestenko al. (2020) in the context of genera- tive networks show that deep neural networks can transform a one-dimensional uniform distribution in way to approximate any wwo-dimensional Lipschitz continuous target density.
Another strand of research related to the questions of this article are works on quantile (or distri- bution) regression, see Koenker (2005) as well as Dabney al. (2018); Tagasovska Lopez-Paz (2019); Fakoor et al. (2021) for recent methods involving neural networks.
DEEP BELIEF NETWORKS
A restricted Boltzmann machine (RBM) is an undirected, probabilistic, graphical model with bipartite vertices that are fully connected with the opposite class. To be more precise, we consider simple graph g (V, E) for which the vertex set V can be partitioned into sets V and H such E
that the edge set is given by E = (s, V. & H}. We call vertices in V visible units; H contains the hidden units. To each of the visible units we associate the state space fy and to the
hidden ones associate QH. We equip G with Gibbs probability measure
(v.A))
T(U, h)
v E (V he (2H)
Z
2

---

Under review as a conference paper at ICLR 2023
(Rz)** + R is chosen such that Z =ff e (f.h) du dh co. Notice that the integral becomes a sum if fyy (resp. QHz is a discrete set. It is customary to identify the RBM with the probability measure
where Ht (Ωv)
An important example are binary-binary RBMs. These are obtained by choosing Sty = Q2 = (0, 1) and
Ht (u, Wh) + (v.b) + (h, DE {0,1}", h {0.
where & {0, 1 and {0, 1 are called biases, and W E Vx is called the weight matrix. We shall write for m, n2 N.
(mR n)) { is a binary-binary RBM with visible and hidden units}.
for the set of binary- RBMs with fixed layer sizes.
The following discrete approximation result is well known, see also Montúfar Ay (2011):
Proposition 1 (Le Roux & Bengio (2008), Theorem 2). Let E N and μ be a probability distribu- lion on {0, 1 1 Let
supp( {u E {0,1)"* p(v) > 0}
be the support of μ. Set supp(u) + 1. Then, for each O, there is a ग B-RBM(m, n) such that
m(v) Σ T(e,h) E (0.11
V (0,1)
deep belief network (DBN) constructed by stacking two RBMs. To be more precise, we now consider a tripartite graph with hidden layers H1 and H2 and visible units V. We assume that the edge set is now given by E = {(,, (2) : E V, E H1, l2 E H2).. The state spaces are now fly IR and Qtg, fH {0, 1}. We think of edges in the graph dependence of the neurons (in the probabilistic sense). The topology of the graph hence shows that the vertices in V and H2 shall be conditionally independent, that is, we require that
plu, h1, h2) = p(u (h)1, h2)
(3)
The joint density of the hidden units p(h1, h2) will be chosen binary-binary RBM.
Let D(Rd) = {f : Rdd + R4 Saa f(x) dx 1} be the set of probability densities on Rd For
D(Rd) and >> O we set
VG. [[19 Pu. Ppoo= =0 (1===0000)))))))))))("))) 1 Rd
P
(4)
Notice that all elements of vg are themselves probability distributions. We fix parental density X D( and choose the conditional density in (3) as p(11 h1) V for each h2 H1. Example The most popular choice of the parental function yp in (4) the d-dimensional standard Gaussian density
1
|x/2
(xx)=
(2T))//2
exp
2
T EER
(5)
Another density considered previous works is the truncated exponential distribution
d
d,e - A,
p(z)=
II
1 - e bide (x,),
x (#1...., Ia) E Rd
(6)
i=1
where b,, A; > 0 for each 1.....d. d.
Similar to (2), collect all DBNs in the set
(d, m) {p is DBN with parental density y, d visible units, hidden
units on the first level, and n hidden units on the second level
3

---

Under review as a conference paper at ICLR 2023
where E ()R") and d.m.n N. We shall not distinguish between the whole DBN and the marginal density of the visible nodes, which is the object we are ultimately interested in, that is, we write
p(u) = Σ ΣE
Σ
p(u, h1, h2)
(7)
MJEH, N2E
In case p DBN,pdd, m,, n.) with yp (I( then also the marginal (7) belongs L*(
After their introduction in Hinton & Salakhutdinov (2006), deep belief networks rose to prominence due to training algorithm developed in Hinton et al. (2006) which addressed the vanishing gradient problem by pre-training deep networks. Instead of naïvely stacking two RBMs the authors consid- ered several such stacked layers and greedily pre-trained the weights over the layers on a contrastive divergence loss. To be more precise, let .M denote the number hidden layers, then, first the visible and the first hidden layer are considered are learnt. In the second step, the weights of the second hidden layer are learnt based on the first hidden layer using Gibbs sampling. This procedure repeats iteratively until all hidden layers are trained. For more details we refer to Fischer & Igel (2014); Ghojogh et al. (2021).
classical RBM and the weights of the first hidden layer
3
MAIN RESULTS
To state the results of this article, we need to introduce three bits of additional notation: Let g E [1, oo]. We declare (x4)) ((Id)) n (P(4). Finally, for (o), let abbreviate the constant
9
1, =max
T, = max
dr
V2
+
(8)
2x
-00
9>
2
π
with the Gamma function [(x) 1000 e--le-t dt, x 0.
The main results of this paper are stated in the following two theorems:
Theorem 3. Let [1, 20) and f, D. ( Then, for each m N, the following quantitative bound holds:
2T 2T
inf (d.mm...
If - - & m min(g.2)
where the constant TY is defined in (8).
While this bound becomes trivial if = the following qualitative approximation result still holds in that case: For any >> 0, there is an M E I such that, for each m M, we can find p E DBN & (d, m, m + 1) satisfying
If 1-pll SE.
Remark 4. Returning to Example we find that 1|ol120 = 1 for the d-dimensional standard normal distribution (5) and
| II
9 (1 e1-e- bid)) (1-
for the truncaled exponential distribution (6). Our bound (9) thus shows that deep belief networks with truncated exponential parental density (for suitable choice of the parameters and X) better approximate the target density f. This especially prevalent for small q. which is the primary case of interest, see Corollary 7 below. For detailed review of the exponential family's properties we refer to Brown (1986).
To state the approximation in the -non, we need to introduce the space of bounded and uni- formly continuous functions:
C. (RA))
lim sup [s(x) ( =0
610 -e-viçs
Notice that any probability density f E D( which is differentiable and has a bounded derivative, belongs C., (R.) since any uniformly continuous and integrable function is bounded.
4

---

Under review as a conference paper at ICLR 2023
Theorem 5. Let f D(R") (R and D (R Then, for any 0, there is an M E N such that, for each m z M. we can find p E DBN (d, m, m - 1) satisfying
If - p E.
Remark 6. The uniform continuity requirement on f in Theorem 5 can actually be relaxed to es- sential uniform continuity, that is, is uniformly continuous except set with zero Lebesgue measure. The most notable example of such function is the uniform distribution f = [1
Another important metric between between probability densities p. q Rdd R4 is the Kullback- Leibler divergence (or relative entropy) defined by
KL(SI19) = =Jpe f(x) log
f(x)
dr
g(x)
(x E Rd g(x) = 0} C for E Rd : f(z) = 0} and KL(f = oc otherwise. From deduce the following quantitative approximation bound in the Kullback-
Theorems 3 and 5
Leibler divergence:
Corollary 7. Let y E (R.)). Let C. Rd be a compact set and f S + R4 be contimoous probability density. Suppose that there is rj O such that both f nj and up 3 n Q. Then there is constant M such that, for each m E N, it holds that
M (8110 + 1lf -11ff(1lia())) (
inf PEDBN, m11
KL(flP)) &
(10)
Tmm
where ||ʃ - ( So [1x)) x(x) dr.
Let us note that any yp E (RR" is intareeit so that the right-hand side of the bound (10)
actually finite. This follows from the interpolation inequality
1 & II Vllsllem,
(11)
(Brezis, 2011, Exercise 4.4).
Remark 8. The first assertion of Theorem 3 and Theorem 5 generalize deep belief networks with additional hidden layers, however, it is still an open question whether (9) can be improved by adding more depth, see also Jalali al. (2019) for analysis of this question in the context of Gaussian mixture models.
Corollary 7 considerably generalizes the results of (Krause et al., 2013, Theorem 7). There, the authors only prove that deep belief networks approximate any density in the closure of the convex hull of a set of probability densities satisfying certain conditions, which appear to be difficult check in practice. That work also does not contain convergence rate. In comparison, our results directly describe the class of admissible target densities and do not rely on the indirect description through the convex hull. Finally, there is an unjustified step in the argument Krause et al., which appears hard to reconcile, see Remark 16 below for details.
PROOFS
This section presents the proofs of Theorems 3, 5 and Corollary 7. As first step, shall establish a couple of preliminary results in the next two subsections.
4.1 L [P-APPROXIMATIO OF FINITE MIXTURES
Given set A C (RR), the convex hull of A is by definition the smallest convex set containing A; in symbols cA))(A). can be shown that
conv(A)= = Σ 0;4; : a (ai,..., an) E Ans a1...., a, E AnneN
5

---

Under review as a conference paper at ICLR 2023
with A. = e Ei=1 = 1), the n-dimensional standard simplex. It is also convenient to introduce the truncated convex hull
m
conv m (A)
0 a (01,...,. amm) E Am a1........... Om E A
for E N so that conv(A) UmEn conv (A). The closed convex hull conv is the smallest closed convex set containing and is straight-frrward to check that it coincides with the closure of conv(A) in the topology of (*
The next result shows that we can approximate any probability density in the truncated convex hull of the set (4) arbitrarily well by DBN with fixed number of hidden units:
Lemma 9. Let oo], i (R 0, and m N. Then, for every f conv (V,) and every ε 0, there deep belief p D (d, m, m 1) such that
f 21 E.
Proof. Since f convm (Vg), there by definition of Am (m).-..,mm) Am and (M)) E (IRd) such that
m a.ppaao
We can think of a = (01,... Om as a probability distribution ā on {0, 13 by declaring
at
if h1 = =
a(h1) = 0, else,
h1 E [0,
where (ei); = di.j. j = 1,..., m, is the 1h unit vector.
Let us fix q E [1, oo] and o By Proposition I there is a e B-RBM(m, m + 1) such that
a(h))
Σ
Th1 h2)
< mo 1
4h E 10.1)m.
(12)
We set
(u))(()) hn = Cis else,
p
10.
and
p(n, b1, h2) p(u) hy (T) E DBN, (d, m, m + 1). This is the desired approximation since
|| - Pl
Σ
12)
|K E
where we used that //y |144 = o I| for each E Rd and each o >
4.2 APPROXIMATION BY CONVOLUTION
Let LR(R), [1, and yp D(Rd). We denote the convolution of f and po by (f * po) px) [ f1)) 11) dp dp= f. f(H))un. dp.
Young's convolution inequality, Young (1912), implics that f Go L9(Rd). In addition, the following approximation result holds, see Appendix A.1 for the proof: Proposition 10. Let E D(RA). Then all of the following hold true:
1. For each g E oo) and each (RR),, have
lim || - f * Po 1. = 0. L
2. If L (RR) (C, then
lim f - f + worl O. d10

---

Under review as a conference paper at ICLR 2023
4.3 APPROXIMATION THEORY IN BANACH SPACES
The second ingredient needed in the proof of Theorem is an abstract result from the geometric the- ory of Banach spaces. To formulate we need introduce the following notion: The Rademacher type of Banach space (X,, || I[x)) the largest number > for which there is a constant C >
such that, for each à N and each f1...., Sk X,
k Σ
E
c
<C>
Σ
holds, where , are i.i.d. Rademacher random variables, that is, P((l =1) =z. It can be shown that for every Banach space.
Example 11. The space (R())) has Rademacher type min(y, 2) for q E [1, oo). The space L (RR) on the other hand has only trivial type t= =1
good reference for the above results on the Rademacher type is (Ledoux & Talagrand, 1991, Section 9.2). The next approximation result and its application to (IRR) will be important below:
Proposition 12 (Donahue et al. (1997). Theorem 2.5), Let (X, (1x) be a Banach space of Rade- macher lype t [1,2]. Let C X and f E conv Suppose that & = supgeA II gllx < co. Then there is constant C only depending on the Banach space (X, || IIx)) such that, for each m E N. we can find an element E conv,, (A) satisfying
CE
|l - 111x < m
(13)
Notice that the bound (13) of course trivial for t 1. Moreover, in Appendix A.2 we provide an example which shows that the convergence rate m optimal.
Corollary 13. Ler C )R & 9 < oc, and suppose that f E conv(A). Ift = supgeA I gllx < 0o, then for all m E N. there is a E conv (A)) such that
S
IV - hell <
m
mintg:27
where T Eg is the constant defined in (8).
Proof.
Owing to Example 11 we are in the regime of Proposition 12. The sharp constant C' T was derived Haagerup (1981).
4.4 PROOF OF THEOREMS 3 AND 5
Before giving the technical details of the proofs, let us provide an overview the strategy:
By Proposition 10 we can approximate the density f D,,((R)) with f * yo up to an error which vanishes as O.
2. Upon showing that f * yo. conv(Vg), Proposition 13 allows us show that for each E > 0 and each m E N, we can pick o > 0 such that
2Y, 1rg1e
inf gEcony (Vg))
|| gll S
m mintu.2]
3. Finally, we employ Lemma 9 to conclude the desired estimate (9).
Lemma 14. Let g E [1, oo], D, (Rd), and ip E D(R)), Then, for each a > 0, we have
f yo E conv(Vg),
with the closure understood with respect the norm -|
7

---

Under review as a conference paper at ICLR 2023
Proof. Let us abbreviate g f Po. We argue by contradiction. Suppose that g & conv conv(Vç). As consequence of the More precisely, there continuous linear function p [*(RR) E such that p(h) < p(g) for all h E conv(V see (Brezis, 2011, Theorem. 1.7). On the other hand, we however have
theorem, 9 is separated from conv(V?) by hyperplane.
p(g) p (S -fee (())p(puo))) du < p(0) / f(0) du pl)),
which is the desired contradiction.
We can now establish the main results of this article:
Proof of Theorems 3 and 5. Let us first assume that q E (1, 0o) and prove the quantitative bound (9). To this end fix & > and m N. We first observe that, by Proposition 10, we can choose 0 sufficiently small such that | - f Po Lo < Employing Lemma 14 and Corollary 13
with A VG. we can find Im E (V such that
|| gmllra < ||s S S + +||s
smller 8
To
sup || Spr MERd
2
minte
For the last term bound
(Je Jpe f(x)(po(y- -Poo 11) dx dy sup HERd
sup || * fo -uollla
#ERd
11olu- x)) s2(v 11)1 dug
$ Jsd
((x) sup HEle
= sup HER"
||1- Puallza < 2115s/2e0
whence
|| - gmllzo < E + 2
1
2Tpll:llle
Finally, Lemma 9 allows us to choose E DBN, (d, m. m+ 1) such that Il9m -plizu < Therefore, we conclude
2T, 2Tg11plle
|| - ppll s
m mintu.27
Since >> 0 was arbitrary, the bound (9) follows.
g = 1 or q = oo, we use the fact that
CONV (A)
U conv, m (A)
mEN
for any subset A of either (RR)) or L (RR), respectively. This implies that, for each 0, we can find m N and 9m E conv, (Vg) such that || PP 9m La & 3- If q cc, we note that uniformly continuous and integrable function is always bounded. Hence, in any case we can apply Proposition 10 to find a > 0 for which | f PollIo & 5. Finally employing Lemma 9 as above, there is p DBN (d, m. m + 1) such that
| 1pll < || f * x llo + ||f - Imllls + I|9m plles E
4.5 KULLBACK LEIBLER APPROXIMATION ON COMPACTS
Let us begin by bounding the Dellbackk-eeibeee divergence in terms of the T,2-norm: Lemma 15 (Zeevi & Meir (1997), Lemma 3.3). Let Q C Rdd, f probability densities. If there is an > O such that both f.g 2 n on Q then
R,, and g Rdd R2 be
kL()llg) < - |f - o
8

---

Under review as a conference paper at ICLR 2023
Proof. We use Jensen's inequality and the elementary fact log & x 1, > 0, obtain
kL(fll9)= ./1 log ((x)) f(x) dx log (ʃ f(x)2 dr
Jn
g(x)
g(z)
/ f())2 dr 11 / (f(x) -(())=))))2 g(x))2
dr & 1
[ 9 1
g(x)
g(x)
Finally, we can prove the approximation bound in Kullback-Leibler divergence:
Proof of Corvllary 7. Extending the target density by zero on IR Q the corollary follows from Theorem 3 upon showing that, for each E N, we can choose the approximation D m, m + 1) in such a way that p > 2 on Q.
To see this, notice that f is uniformly continuous since Ω is compact. Hence, Theorem allows us to pick an M E N such that, for each m > M, there is a Pm p m, m + 1) with |[ - Pm = & 7. In particular, each of these DBNs satisfies Pm > ! on Ω. Consequently, by Lemma 15 we obtain
2 17m
inf PE PEDBN, (1)
kL(fIlp) <
Vm > M.
(14)
A crude upper bound Inf zero weights and biases in (1) as well p(u) hy) pp for each ha {0, 1)mh m in (3). Hence, the visible units of the DBN have density sp. This gives
(T(fllp) for m M can be obtained choosing both
kL((Ilp) & KT(SIlØ) & I15 +112210)
inf PEDBN (d.
vm= 1......... ..... M- (15)
again by Lemma 15. Finally, combining (14) and (15) get the required estimate:
M (SlPliE + |lf - -
inf PEDNN, m11
KL(FIIp) &
TIm
Remark 16. Our strategy of the proof of the Kullback-Leibler approximation in Corollary 7 through Lemma 15 differs from the one employed in (Krause al., 2013, Theorem 7). There, the authors built on the results of Li & Barron (1999) and in the course of their argument claim that the following statement holds true:
Let fm. f f 1--- +R R., m E N. be probability densities If (L + as m oo, then fm - f in the norm || I
compact set Ω C Rd with fm:
This, however, does not hold as we illustrate by simple counterexample presented in Appendix A.3.
5 CONCLUSION
We investigated the approximation capabilities of deep belief networks with two binary hidden layers of sizes and m 1, respectively, and real-valued visible units. We showed that, under minimal regularity requirements on the parental density as well as the target density f. these networks are universal approximators the strong L and Bellback--eibbee distances m - oc. Moreover, gave sharp quantitative bounds the approximation error. We emphasize that the convergence rate the number of hidden units independent of the choice of the parental density.
Our results apply to virtually all practically relevant examples thereby theoretically underpinning the tremendous empirical success of DBN architectures we have seen over the last couple years. As we alluded to in Remark 4, the frequently made choice of Gaussian parental density does not provide the theoretically optimal DBN approximation of a given target density. Since, in practice, the choice of parental density cannot solely be determined from approximation standpoint, but also the difficulty of the training of the resulting networks needs to be considered, it is interesting to further empirically study the choice of parental density both artificial and real-world datasets.

---

Under review as a conference paper at ICLR 2023
REFERENCES
A. M. Abdel-Zaher and A. M. Eldeib. Breast cancer classification using deep belief networks. Expert Systems with Applications, 46:139-144, 2016.
B. Bailey and Telgarsky. Size-noise tradeoffs in generative networks. Advances in Neural Information Processing Systems, 31, 2018.
A. R. Barron. Universal approximation bounds for superpositions of sigmoidal function. IEEE Transactions on Information Theory, 39(3):930-995 1993.
A. R. Barron. Approximation and estimation bounds for artificial neural networks. Machine learn- ing, 14(1):155-133, 1994.
Y. Bengio and Delalleau. On the expressive power of deep architectures. International con- ference algorithmic learning theory, pp. 18-36. Springer, 2011.
H. Brezis. Functional analysis, Sobolev spaces and partial differential equations. Universitext. Springer, New York, 2011.
L. D. Brown. Fundamentals of statistical exponential families: with applications in statistical deci- sion theory. IMS, 1986.
M. Burger and A. Neubauer. Error bounds for approximation with neural networks. Journal of Approximation Theory, 12:235-250. 10 2001.
R. Cao, D. Bhattacharya, Hou, and J. Cheng. Deepqa: improving the estimation of single protein model quality with deep belief networks. BMC Bioinformatics, 17(1):1-9, 2016.
G. Celcux. EM methods for finite mixtures. In Handbook of mixture analysis. Chapman & Hall/CRC Handb. Mod. Stat. Methods, pp. 21-39. CRC Press, Boca Raton, FL, 2019.
Chen and Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its applications to dynamic systems. IEEE Transactions on Neural Networks, pp. 911-917, 1995.
G. V. Cybenko. Approximation by superpositions sigmoidal function. Mathematics of Control, Signals and Systems, 2:303-314, 1989.
W. Dabney, M. Rowland, M. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
M. J. Donahue, C. Darken, L. Gurvits, and E. Sontag. Rates of Convex Approximation Non- Hilbert Spaces. Constructive Approximation, 13(2):187-220, 1997.
B. S. Everitt and D. J. Hand. Finite mixture distributions. Monographs Applied Probability and Statistics. Chapman & Hall, London-New York, 1981. ISBN 0-412-22420-8.
R. Fakoor, T. Kim, J. Mueller, A. J. Smola, and R. J. Tibshirani. Flexible model aggregation for quantile regression. arXiv preprint arXiv:21033 103.00083, 2021.
A. Fischer and C. Igel. An introduction to restricted Boltzmann machines. In Theroamerican Congress Pattern Recognition, pp. 14-36. Springer, 2012.
A. Fischer and C. Igel. Training restricted Boltzmann machines: An introduction. Patlern Recogni- tion, 47(1):25-39 2014.
G. B. Folland. Real analysis. Pure and Applied Mathematics (New York). John Wiley & Sons, Inc., New York, second edition, 1999.
B. Ghojogh, A. Ghodsi, F. Karray, and M. Crowley. Restricted Boltzmann machine and deep belief' network: Tutorial and survey. arXiv preprint arXiv:2107.12511, 2021.
U. Haagerup. The best constants in the Khintchine inequality. Studia Math., 70(3)2211-28,, 1981.
10

---

Under review as a conference paper at ICLR 2023
G. E. Hinton. Deep belief networks. Scholarpedia, 4(5):5947, 2009.
G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313:504-507, 2006.
G. E. Hinton, S. Osindero, and Y.W.. Teh. A fast learning algorithm for deep helief nets. Neural computation, 1877):157-1554, 2006.
K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approxi- mators. Neural networks, 2(5):359-366, 1989.
Y. Huang, A. Panahi, H. Krim, Y. Yu, and S. L. Smith. Deep adversarial belief networks. arXiv preprint arXiv: 1e09.9999000 1et.:09999043444 06134, 2019.
Jalali, C. Nuzman, and I. Sanice. Efficient deep approximation of gmms. Advances in Neural Information Processing Systems, 32, 2019.
M. Jiang, Y. Liang, X. Feng, X. Fan, Z. Pei, Y. Xue, and R. Guan. Text classification based on deep belief network and softmax regression. Neural Computing and Applications, 2961):61-70, 2018.
J. Johnson. Deep, skinny neural networks are not universal approximators. In International Confer- ence on Learning Representations, 2018.
L. K. Jones. A simple lemma on greedy approximation in hilbert space and convergence rates for projection pursuit regression and neural network training. The Annals of Statistics, pp. 608-613, 1992.
Kamada and Ichimura. An adaptive learning method of deep belief network by layer generation algorithm. In 2016 IEEE Region 10 Conference (TENCON), pp. 2967-2970. IEEE, 2016.
S. Kamada and T. Ichimura. An object detection by using adaptive structural learning of deep belief network. 2019 International joint conference on neural networks (IJCNN), pp. 1-8. IEEE, 2019.
R. Koenker. Quantile regression, volume 38 of Econometric Society Monographs. Cambridge University Press, Cambridge, 2005.
O. Krause, A. Fischer, T. Glasmachers, and C. Igel. Approximation properties of dbns with binary hidden units and real-valued visible units. International Conference on Machine Learning, pp- 419-426, 2013.
N. Le Roux and Y. Bengio. Representational power of restricted Boltzmann machines and deep belief networks. Neural Computation, 20(6):633-1169, 2008
N. Le Roux and Y. Bengio. Deep belief networks are compact universal approximators. Neural computation, 22:2192-2201 2010.
M. Ledoux and M. Talagrand. Probability in Banach spaces, volume 23 of Ergebnisse der Mathe- matik und ihrer Grenzgebiete (3). Springer-Verlag Berlin, 1991.
M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken. Multilayer feedforward networks with a non- polynomial activation function can approximate any function. Neural networks, 6(6):861-867, 1993.
J. Q. Li and A. R. Barron. Mixture density estimation. In NIPS, volume 12, pp. 279-285. 1999.
M. Liang. Z. Li, T. Chen, and J. Zeng. Integrative data analysis of multi-platform cancer data with multimodal deep learning approach. IEEE/ACM transactions on computational biology and bioinformatics, 12(4):928-9))
2014.
P. Luo, Y. Li, L.-P. Tian, and F.-X. Wu. Enhancing the prediction of discasc gene associations with multimodal deep learning. Bioinformatics, 35(19):3735-374, 2019.
G. McLachlan and K. E. Basford. Mixture models, volume 84 of Statistics: Textbooks and Mono- graphs. Marcel Dekker, Inc., New York, 1988.
11

---

Under review as a conference paper at ICLR 2023
G. McLachlan and D. Peel. Finite mixture models. Wiley Series in Probability and Statisties: Applied Probability and Statistics. Wiley-Imterseience, New York, 2000.
T. Merkh and G. Montúfar. Stochastic feedforward neural networks: Universal approximation. arXiv preprint FreXiv::910009900 2019.
G. Montúfar. Deep narrow Boltzmann machines are universal approximators. 2014.
G. Montúfar. Universal approximation of markov kernels by shallow stochastic feedforward net- works. arXiv preprini arXiv: 1503.00211 2015.
G. Montúfar. Restricted Boltzmann machines: Introduction and review. In Information Geometry and Its Applications IV, pp. 75-115. Springer, 2016.
G. Montúfar and N. Ay. Refinements of universal approximation results for deep belief networks and restricted Boltzmann machines. Neural computation, 2365;1306-1319, 2011.
G. Montúfar and J. Morton, Discrete restricted Boltzmann machines. Mach. Learn. Res., 16(1): 653-672, 2015.
H. D. Nguyen and G. McLachlan. On approximations via convolution-defined mixture models. Communications Statistics Theory and Methods, 48(16):3945-399) 3955, 2019.
T. T. Nguyen, H. D. Nguyen, F. Chamroukhi, and G. J. McLachlan. Approximation by finite mix- tures of continuous density functions that vanish at infinity. Cogent Mathematics & Statistics, (1):1750861, 2020.
R. Pascanu, G. Montúfar, and Y. Bengio. On the number of response regions of deep feed forward networks with piece-wise linear activations. In International Conference on Learning Represen- lutions, 2014.
D. Perekrestenko, S. Müller, and H. Böleskei. Constructive universal high-dimensional distribution generation through deep relu networks. In International Conference on Machine Learning, pp- 7610-7619. PMLR, 2020.
C. Robert and K. L. Mengersen. Exact Bayesian analysis of mixtures. In Mixtures: estimation and applications, Wiley Ser. Probab. Stat., pp. 241-254, Wiley, Chichester, 2011.
F. Shen, J. Chao, and J. Zhao. Forecasting exchange rate using deep belief networks and conjugate gradient method. Neurocomputing, 167:243-253. 2015.
Smolensky, Information processing in dynamical systems: foundations of harmony theory. In Par- allel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, pp. 181. 1986.
I. Sutskever and G. E. Hinton. Deep, narrow sigmoid belief networks are universal approximators. 2636, 2008.
Neural computation,
N. Tagasovska and Lopez-Paz. Single-mon model uncertainties for deep learning. Advances Neural Information Processing Systems, 32, 2019.
D. M. Titterington, A. F. M. Smith, and U. E. Makov. Statistical analysis of finite mixture distribu- tions. Wiley Series in Probability and Mathematical Statistics: Applied Probability and Statistics. John Wiley & Sons, Ltd., Chichester, 1985. ISBN 0-471-90763-4.
Y. Wang and J. Zeng. Predicting drug-target interactions using restricted Boltzmann machines. Bioinformatics, 29113)) 2013.
W. H. Young. On the multiplication of successions of Fourier constants. Proceedings of the Royal Society of London. Series A, Containing Papers of Mathematical and Physical Character, 87 (596):331-33) 1912.
A. J. Zeevi and R. Meir. Density estimation through convex combinations of densities: Approxima- tion and estimation bounds. Neural Networks, 10(1):99-109, 1997.
12

---

Under review as a conference paper at ICLR 2023
A DETAILS OF THE MATHEMATICAL RESULTS
This appendix provides further details of the mathematical results used in the main text. More specifically, we provide
1. the proof of Proposition 10,
2. detailed proof of Proposition 12 for Hilbert spaces as well an example showing that its approximation rate optimal in general, and
3. the construction of explicit counterexample to the statement discussed in Remark 16.
A.1 PROOF OF PROPOSITION 10
Proof. Item is well known, see e.g. (Folland, 1999, Theorem 8.14). For 2 fix 8 0. By uniform continuity of f, we can find a 5 > 0 such that
sup [f(x) f(x 1) & 1n
Vx E
(16)
partieular, we obtain
f(x) (x))
< / (x)))((()) f(x - -n)
& ((mbon (o(w)) - f(x - dp + ( Jamken (o(k)))(()) f(x - u)) du
e 2' [ [(MD2 (p) duc
< <1|lS|I>>>
where we applied the uniform continuity estimate (16) to the second integral. Since
J(mIson (H( du S [[mDze) p(u) du + O 0,
choose do such that || (ʃ x & for all (0, oo)- This completes the
proof.
A.2 DETAILS ON PROPOSITION 12
While the proof of Proposition 12 for general Banach space rather technical, find it instructive to present the simplified argument for Hilbert space. Our proof is inspired by Jones (1992), see also Barron (1994).
Let (X, l1x) be a Hilbert space. Let C X and f E conv Suppose that & suPgEA Ilf glix < 00. Then, for each m E N. can find an element E conv (A) satisfying
Proposition 17.
Ilf -llxx & m
(17)
Proof. We proceed by induction on m e N. The base m is trivial, so we can assume that the statement holds for m > 1. Let us declare
Em+1
inf DEconnmiill
|| - gllx-
13

---

Under review as a conference paper at ICLR 2023
By the induction hypothesis, may assume that Em attaining this bound. Consequently, we get
_ and find h conv mm
A(S 9) -1- - -1i) 11)
inf AE gEA
m+
inf 1
Aal1S gl1k + +2(11 2) (5 -. f 11xx + +1 2)9115 Mixx
(18)
gEA
inf
xRE² + 21(1 A) inf (f - 9. - -1)x (1- -1-1)
XE(O,1]
gEA
We claim that
-f 9. x1x gEA
inf
(19)
To this, let us fix an 0 and observe that, since E conv(A), the Bauchy-Schwazz inequality implies that there must be a finite convex combination of elements in A satisfying
k
k
a (f - f - h) =
Σ a f - h b)aE
E
In particular, the inequality (f- ai ai, was arbitrary, we have established (19).
f f- h) < holds for at least one vector at E A. Since
Inserting (19) in (18), we arrive at
2²=2 2
1262 (1 x> AE(0,1]
inf
=2
where the last step follows by chosing
=² 22 +
E [0, 1]. Finally, recalling the induction hypoth-
esis Em & V.n. we conclude
2
E
m
m
+
This establishes (17) for m 1 and the induction is complete.
Returning the original statement of Proposition 12 for general Banach space, the next example shows that its convergence rate is optimal in general:
Example 18. For p E (1,2) let us consider the Banach space (RR of p-summable real-valued sequences, that is, (an)neN C IR belongs to (R) iff
|alle
Σ
Jan IP
< 00.
It can be shown that this Banach space has Rademacher type = p. Let A be the formed of the standard basis vectors:
A= 11000,0,00...10011,,0,......
Choosing f = O, we find that
m
||ʃ hellr inf kEcouv (A)
inf
(an Eaa
The optimum on the right-hand side is attained by choosing
Om
so that
1
inf ||ʃ - lollex hecouv
771
14

---

Under review as a conference paper at ICLR 2023
A.3 CONSTRUCTION OF THE COUNTHREXAMPLE IN REMARK 16
Let Ω = [0, 1] and consider the sequence of probability densities given by
fa( = Cm I mE + 2)) m M,N,
where Cm = (1 - 1/(8m))) is chosen such that fo (z) dx [0.1)()) = f(x pointwise on (0, 1]. On the other hand, holds that
Then we have fz) 7
sup [m(()) - [()) (0m( s(0) = 2 =
Vm E N.
Consequently, fm does not converge uniformly to f.
Nevertheless, is strwight-forward to check that || - 10.1 12 + and since fm f > 1/2 on U we have KL [[] - 0 as m + oo by Lemma 15.
15


Reviews:
[{"review_id": "ywuh6qxKhZz", "reviewer": "Reviewer_heng", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper looks at the approximation capabilities of a DBN with 2 layers in which the first layer is a has binary states (both hidden and visible) and the second layer has continuous visible state. Further, it is assumed that the conditional distributions for the last layer all come from the same parental distribution (that is, they are just shifts and rescalings of each other). Under such an assumption, we can think of the DBN as mixture model (such as a Gaussian Mixture model with $\\sigma I$ as the covariances when the parental distribution is Gaussian). \n\nIn this set up the paper shows that as the number of hidden nodes goes to infinity, the model can approximate arbitrarily well any measure that is absolutely continuous with respect to Lebesgue measure (i.e. measures that have distributions with respect to the Lebesgue measure.) That is, as the number of mixtures factors increases we can approximately any density. \n\nIn some cases the paper provides error rates. \n\n", "summary_of_the_review": "In summary, I think this is a well written paper and with interesting and new results. ", "strength_and_weaknesses": "**Strengths**\n---\n\n1) I think the major strength of the paper are the error rates that it presents. In particular, I think even for Gaussian Mixture I dont the optimal error rate is known. \n\n2) I think writing is clear and proofs are easy to follow and use mostly standard analysis tools. \n\n**Weaknesses**\n---\n\n1) I think a discussion to prior theoretical work on approximations by mixture models is missing. Specifically in the case when we think of the special cases of mixtures of Gaussians and mixtures of exponential families. Significant prior work has been done. In this regard, I think the universality is known (please correct me if I am wrong). However, the strength of the this paper are the error rates. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "rsjY0EkCn4-", "reviewer": "Reviewer_F7vv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript investigates the approximation error of classic deep belief networks (DBNs), in particular DBNs with two hidden layers of size m and m+1, respectively. It is demonstrated that, under both  L^q-norm and Kullback-Leibler divergence, DBNs are universal approximators. Moreover,  as claimed by the author, sharp bounds are obtained for the approximation error. ", "summary_of_the_review": "This manuscript is theoretically studied the approximation error of DBNs with   two binary hidden layers of sizes m and m + 1, respectively. In particular, under both L^q and Kullback-Leibler divergence metrics,  DBNs are proven to be universal approximators and the bounds of the approximation errors have been obtained.  It is a solid work with theoretical contributions. My main concern lies in the restriction of the sizes of the hidden layers to be m and m + 1, as well as a lack of (empirical) evidence of the sharpness of the obtained bounds. ", "strength_and_weaknesses": "Strengths:\n\n1. This is a solid theoretical work on the analysis of approximation error of DBN, which has been an open problem. \n\n2. Two metrics, L^q-norm and KL deivergence, are considered. \n\nWeaknesses:\n\n1. It seems that the main results of this manuscript are restricted only to the case of DBNs with two hidden layers of size m and m+1 respectively. Can the results still hold for general size m and n for two layers? If not, why? \n\n2. The authors stated that, e.g., in the abstract,  \"we establish a sharp quantitative bounds on the approximation error in terms of the number of hidden units \". How to verify the sharpness of the obtained bounds? I did not see any supportive results to show that the obtained bounds are sharp. Can the authors add more details or explanations on this point? I would be very nice if some empirical evidence is provided. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "l-c3Gp_ju0_", "reviewer": "Reviewer_V3vX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider approximation of probability density functions by deep belief networks with binary hidden units. Convergence and rates of approximation are provided with respect to the L_q norm and K-L divergence. ", "summary_of_the_review": "The topic of approximation by deep belief networks is interesting. Rates of convergence given by applying the Rademacher index of the L_q spaces might have applications in some other problems. But a large literature of distribution regression is missing in the paper. Using convolutions for approximations without involving regularity of the approximated function is rather shallow. ", "strength_and_weaknesses": "The topic of approximation by deep belief networks is interesting. Rates of convergence are given by applying the Rademacher index of the L_q spaces. There have been many results in the literature of distribution regression, on learning of probability density functions. The authors do not provide any comparisons with the existing methods for approximating probability density functions. Using convolutions for approximations is fine, but regularity of the approximated function is not used. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "WGEfL445shJ", "reviewer": "Reviewer_atRJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A purely theoretical contribution in which the authors present a series of approximation results describing which densities can be well-approximated by DBNs.  Additionally, they provide quantitative error bounds for both Lp based norms and the KL divergence.", "summary_of_the_review": "An interesting theoretical contribution that precisely describes classes of probability densities that can be well approximated by DBNs.  ", "strength_and_weaknesses": "Strengths:  Fills gaps in the theoretical literature on what densities can be approximated by DBNs.\n\nWeaknesses:  Proofs are a bit terse (as is necessitated by space).  Doesn't contain any experimental evaluation, but I don't think that is necessary for papers of this type.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "29bMgcdsM0", "reviewer": "Reviewer_F3UE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proves univeral approximation theorems for deep belief networks when the targeted distribution is continuous, with quantitative bounds for $L^p$ space with $p>1$. The proof idea is to make use of the efficient approximation of convex hull, each item is represented in the first layer, while the second layer is used to approximate the convex combination utilizing previous results for the discrete case.", "summary_of_the_review": "The results are new and solid, written in a mathematically clean way. I recommend accept.", "strength_and_weaknesses": "Strength:\n\nThe results are new and solid, generalizing previous results for the discrete case to the continuous case.\n\nThe proof idea is simple yet effective, through which this paper solves the targeted problem in a clean way.\n\nWeaknesses:\n\nThe result is well-expected due to numerous analogical universal approxiamtion results for neural networks, and similar techniques have appeared in that literature.\n\nWhat's the dependence of $M$ on $\\epsilon$ in theorem 3?\n\nThe analysis is based on some technical propositions from previous works without proof. Adding more explanation/proof idea/easy example to these introduced propositions would make this paper more readable. For instance, proposition 11 could benefit from explaining the special case of $L^2$, which admits a short proof because it's a Hilbert space.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject