Title: On the Universal Approximation Properties of Deep Neural Networks using MAM Neurons
Venue: NeurIPS 2023
Accepted: no
Keywords: universal approximation theory

multiply-and-max/min neurons
Abstract: As Deep Neural Networks (DNNs) are trained to perform tasks of increasing complexity, their size grows, presenting several challenges when it comes to deploying them on edge devices that have limited resources. To cope with this, a recently proposed approach hinges on substituting the classical Multiply-and-Accumulate (MAC) neurons in the hidden layers of a DNN with other neurons called Multiply-And-Max/min (MAM) whose selective behaviour helps identifying important interconnections and allows extremely aggressive pruning. Hybrid structures with MAC and MAM neurons promise a reduction in the number of interconnections that outperforms what can be achieved with MAC-only structures by more than an order of magnitude. However, by now, the lack of any theoretical demonstration of their ability to work as universal approximators limits their diffusion. 
Here, we take a first step in the theoretical characterization of the capabilities of MAM\&MAC networks. In details, we prove two theorems that confirm that they are universal approximators providing that two hidden MAM layers are followed either by a MAC neuron without nonlinearity or by a normalized variant of the same. Approximation quality is measured either in terms of the first-order $L^p$ Sobolev norm or by the $L^\infty$ norm.

Abstract

As Deep Neural Networks (DNNs) are trained to perform tasks of increasing complexity, their size grows, presenting several challenges when it comes to deploying them on edge devices that have limited resources. To cope with this, a recently proposed approach hinges on substituting the classical Multiply-andAccumulate (MAC) neurons in the hidden layers of a DNN with other neurons called Multiply-And-Max/min (MAM) whose selective behaviour helps identifying important interconnections and allows extremely aggressive pruning. Hybrid structures with MAC and MAM neurons promise a reduction in the number of interconnections that outperforms what can be achieved with MAC-only structures by more than an order of magnitude. However, by now, the lack of any theoretical demonstration of their ability to work as universal approximators limits their diffusion. Here, we take a first step in the theoretical characterization of the capabilities of MAM&MAC networks. In details, we prove two theorems that confirm that they are universal approximators providing that two hidden MAM layers are followed either by a MAC neuron without nonlinearity or by a normalized variant of the same. Approximation quality is measured either in terms of the firstorder L p Sobolev norm or by the L ∞ norm.

1 Introduction

Deep Neural Networks (DNNs) solve complex tasks leveraging a massive number of trainable parameters. Yet, thanks to the recent increasing interest in mobile Artificial Intelligence, there has been a growing emphasis on designing lightweight structures able to run on devices with constrained resources. This can be obtained by removing parameters that do not appreciably influence performance by means of one of the many pruning techniques that have been proposed. Some approaches entail removing, in a single shot, individual interconnections or entire neurons once the DNN has been trained, while others methods are applied iteratively, and require multiple rounds of training. These techniques eliminate interconnections but do not alter the underlying Multiply-andACcumulate (MAC) paradigm that governs the neuron's inner functioning.

In [1, 2], the authors address the challenge of designing neural networks that can have a smaller 28 memory footprint presenting a novel neuron model based on the Multiply-And-Max/min (MAM) 29 paradigm that can be substituted to classical MAC neurons in the hidden layers of a DNN to 30 allow a more aggressive pruning of interconnections, while substantially preserving the network 31 performance. In a standard MAC-based neuron, inputs are modulated independently of each other 32 through multiplication with their respective weights, and the resulting products are then summed into 33 a single quantity. As MAC neurons, MAM neurons multiply each input by a weight but then only the 34 maximum and the minimum quantity of the products are summed together. 35

In formulas, if v 1 , v 2 , . . . are the inputs after being multiplied by their respective weights, the output 36 u of a MAM neuron is 37

where b is the bias and [⋅] + = max { 0 , ⋅} represents the nowadays common ReLU nonlinearity. 38

It is shown empirically that, starting from an architecture originally designed using MAC neurons, one 39 may substitute them with MAM neurons in several hidden layers and use a proper training strategy to 40 achieve the same performances as the corresponding MAC-only network. Yet, in the resulting hybrid 41 network, one may leverage the extremely selective behaviour of min and max operations to reduce 42 very aggressively the number of weights. MAM neurons can be pruned with almost every technique 43 proposed in the literature with little to no modifications. As a motivating example, Table 1 reports 44 some of the results described in [1] showing cases in which, once the quality level is set (in this case 45 to 3% less accuracy than the original non-pruned network), MAM neuron substitution, retraining 46 and pruning reduce the number of weights 1 to 2 orders of magnitude more than what is obtained 47 by pruning the original MAC-only network. Moreover, these neurons can also be pruned iteratively 48 requiring less training iterations to guarantee a given accuracy compared to standard MAC neurons. 49

Though the equivalence between MAC-only and MAM&MAC networks has been demonstrated in 50 practice, a change in the model of some neurons opens the problem of the abstract capability of such 51 hybrid architectures. This contribution is a step forward in clarifying that, despite the locally different 52 input-output relationships, also hybrid MAM&MAC networks enjoy some universal approximation 53 capabilities analogous to those of the MAC-only networks. 54

55

1.1 Brief background on universal approximation properties

The development of models with universal approximation properties has been a significant break56 through in many fields of science and engineering. In 1989 [3] proved that a network with a single 57 hidden layer could approximate any continuous function, given enough hidden neurons. Some years 58 later, [4] and [5] showed that also fuzzy systems could approximate any continuous function to 59 arbitrary accuracy. These works were later extended to multiple inputs and outputs, demonstrating 60 the universal approximation properties of fuzzy systems more broadly ([6, 7]). In the following 61 years, a large number of researchers have studied the universal approximation properties of neural 62 networks with MAC neurons in the case of bounded depth and arbitrary width ([8, 9]), bounded width 63 and arbitrary depth ([10, 11, 12]) and bounded width and depth ([13, 14]). In the recent work [15], 64 authors obtained the optimal minimum width bound of a neural network with arbitrary depth to retain 65 universal approximation capabilities. 66

The research in this field is still very active and aims at proving the universal approximation capa67 bilities of networks with different architectural or computational paradigm choices, such as deep 68 convolutional neural networks [16], dropout neural networks [17], networks representing probability 69 distributions [18] and spiking neural networks [19]. 70

2 Mathematical model 71

We indicate with L(⋅) a fully connected layer in which all neurons are based on the MAM paradigm 72 (1). We consider networks with N inputs collected in the vector x = ( x 1 , . . . , x N ) , two MAM hidden 73 layers producing a vector z ( x ) = ( z 1 ( x ) , z 2 ( x ) , . . . ) = L ′′ (L ′ ( x )) and a single output Z ( x ) ∈ R 74 produced by a final layer that computes either the normalized linear combination 75

We normalize the input domain by assuming x i ∈ X = [ 0 , 1 ] for i = 1 , . . . , N and indicate with Z ∗ 77 the family of functions in (2) while with Z the analogous family of functions in (3). Smoothness 78 conditions on our target functions f ∶ X N ↦ R is formalized by assuming that they belong to 79 C d ( X N ) , i.e., that their d -th order derivatives are continuous. Distances between functions are 80 measured by means of the norms defined as 81

with k = { 0 , 1 } and p ≥ 1 . 82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

3 Main results

Within the above framework, we prove two theorems that describe the universal approximation properties of DNNs using MAM neurons in the hidden layers.

Theorem 1. For any function f ∈ C 0 ( X N ) and any prescribed level of tolerance ϵ > 0 , there is a Z ∈ Z ∗ such that ∥ f -Z ∥ 0 , ∞ ≤ ϵ .

Theorem 2. For any function f ∈ C 2 ( X N ) , any prescribed level of tolerance ϵ > 0 and finite p ≥ 1 , there is a Z ∈ Z such that ∥ f -Z ∥ 1 ,p ≤ ϵ .

The proofs of both theorems are reported in Section 6 and are constructive. In particular, subnetworks in the cascade z ( x ) = L ′′ (L ′ ( x )) are identified and programmed to make each z k ( x ) a weakly unimodal piecewise-linear function of the inputs, whose maximum is 1 and is reached in a hyperrectangular subset of the domain, while the function vanishes for points far from the center of that hyper-rectangle. The shapes and positions of these functions can then be designed along with the values of the weights c k so that their combination by means of either (2) or (3) is capable of approximating the target function arbitrarily well as measured either by ∥⋅∥ 1 ,p or ∥⋅∥ 0 , ∞ .

4 Examples

Figure 1 proposes a visual representation of the constructions behind Theorem 1 and Theorem 2 for 98 N = 2 . From left to right, we report the target function f ∶ X 2 → R 99

and its approximation Z ∈ Z ∗ implied by the proof of Theorem 1 and its approximation X ∈ Z 100 implied by the proof of Theorem 2. In both cases the parameter n used in Section 6 is set to n = 7 . 101

or the linear combination 76

102

103

104

105

106

107

108

109

110

111

112

113

114

115

where 116

x

1

x

1

f

(

x

1

, x

2

)

Z

(

x

1

, x

2

)

X

(

x

1

, x

2

)

x

1

x

2

x

2

x

2

1.0

0.5

0.0

3

0.0

0.5

1.0

1.0

0.5

0'0

5

3

2

0.0

0.5

1.0

1.0

0.5

0.0

5

4

2

0.0

0.5

1.0

5 Limitations

Theorem 1 and Theorem 2 rely on networks in which constraints are put neither on the layer width nor on the total number of neurons. Hence, despite proving universal approximation capabilities, they do not imply efficient approximation. Yet, such theoretical limitation is never strongly experienced in practice, since MAM networks are able to guarantee acceptable performance in real use cases. Nevertheless, a deeper look at universal approximation aimed at meeting efficiency will be the focus of future analysis.

6 Network construction and proofs of Theorems

6.1 Network construction

The aim of this subsection is to show that our network can be programmed to make the outputs of the second hidden layer specific weakly unimodal piecewise-linear functions z k ( x ) of the inputs.

Lemma 1. Let z be any of the outputs of the second hidden layer. For N > 1 and any choice of the quantities ω 1 , . . . , ω N ∈ [ 0 , 1 ] , l 1 , . . . , l N ≥ 0 , δ L 1 , . . . , δ L N ≥ 0 , and δ R 1 , . . . , δ R N ≥ 0 , the two MAM hidden layers can be programmed to yield

Proof of Lemma 1. We assume that neurons in the first hidden layer come in pairs 117 ( y L 1 , y R 1 , y L 2 , y R 2 , . . . ) = L ′ ( x ) and the output of a pair depends on only one of the inputs. 118

Without any loss of generality, we assume that y L i and y R i depend only on x i for i = 1 , . . . , N while all 119 the other N -1 input weights are set to 0 . The other outputs of the first hidden layer are involved in 120 the computation of the outputs of the second hidden layer further to the z we are considering. 121

For y L i the non-null input weight is equal to -1 / δ L i and the bias is ( ω i -l i ) / δ L i , while for y R i the non-null 122 input weight is equal to 1 / δ R i and bias is (-ω i -l i ) / δ R i . By recalling (1) one gets 123

In the second hidden layer, the neuron computing the z we consider has all input weights equal to 0 124 but those connecting to y L 1 , y R 1 , . . . , y L N , y R N . Non-null input weights are equal to -1 and the bias is 1 125 so that 126

δ

R

1

δ

L

2

x

1

z

ω

(

x

1

, x

2

)

x

2

2

/lscript

2

δ

R

2

2

/lscript

1

δ

L

1

(

ω

1

, ω

2

)

1.0

1.0

0.5

0.0

0.0

0.5

0.5

1.0

0.0

1.0

0.8

0.6

0.4

0.2

0.0

0.0

0.2

0.4

0.6

0.8

1.0

Considering the last expression, note that, if x i ≥ ω i then y R i ≥ 0 and y L i = 0 while, if x i < ω i then 127 y R i = 0 and y L i ≥ 0 . Hence, without loss of generality, we may assume that x i ≥ ω i for i = 1 , . . . , N , 128 being all other cases a variation of this one by suitable symmetry and scaling. With this, y L i = 0 for 129 i = 1 , . . . , N and (8) becomes 130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

that is equivalent to the thesis.

To interpret Lemma 1 note that ∆ ( x ) is a scaled measure of how far the input vector x is from the hyper-rectangle centered at ω = ( ω 1 , . . . , ω N ) with sides 2 l 1 , . . . , 2 l N . Hence, z ( x ) is maximum and equal to 1 if x belongs to such a hyper-rectangle and has a piecewise-linear decreasing profile when x gets further from ω . Figure 2 reports an example of a z ( x ) when N = 2 .

In the following, we will assume that each neuron in the second hidden layer matches a whole subnetwork as implied by Lemma 1. With this, we may re-index the outputs of the second hidden layer as z ω ( x ) associating each of them with the center of the hyper-rectangle in which z ω ( x ) = 1 . The same is done with the corresponding weights c ω in the output layers.

6.2 Universal approximation properties with normalized linear output neuron

Given a positive integer n , define Ω = { 0 , 1 n , 2 n , . . . , 1 } N and include in the two hidden layers all the subnetworks implied by Lemma 1 to implement the function z ω ( x ) for each ω ∈ Ω .

In each of these subnetworks set δ L i = δ R i = δ = 1 / n for i = 1 , . . . , N and l i = 0 for i = 1 , . . . , N .

With this, z ω ( x ) is and ( N + 1 ) -dimensional pyramid whose base is an N -dimensional hypercube with sides of length 2 δ and center in ω .

Proof of Theorem 1. Note first that for any given x ∈ X N , only a limited number of functions z ω ( x ) are not null. In particular, if k i = ⌊ nx i ⌋ for i = 1 , . . . , N is the largest integer not exceeding nx i , then z ω ( x ) > 0 only if ω belongs to the set Ω x = { k 1 δ, ( k 1 + 1 ) δ }×⋅⋅⋅ × { k N δ, ( k N + 1 ) δ } that contains the 2 N corners of the N -dimensional hypercube C x = [ k 1 δ, ( k 1 + 1 ) δ ] × ⋅ ⋅ ⋅ × [ k N δ, ( k N + 1 ) δ ] . Hence, we may evaluate Z ( x ) focusing on functions z ω ( x ) with ω ∈ Ω x .

for i, j = 1 , . . . , N .

that are such that ∑ ω ∈ Ω ζ ω ( x ) = ∑ ω ∈ Ω x ζ ω ( x ) = 1 for any x ∈ X N , and set c ω = f ( ω ) for each 152 ω ∈ Ω . 153

The error ∥ f ( x ) -Z ( x )∥ 0 , ∞ in Theorem 1 can be written as 154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

To give some intuitive grounding to the above definitions, Figure 3 reports example profiles for 4 172 output functions z ω 1 -, z ω 1 + , z ω 2 -, z ω 2 + with N = 2 . 173

Given a center ω , the same quantities δ and ℓ allow to define the two domain subsets 174

as well as X ω = X ∎ ω ∪ X /uni25FB ω . 175

Since f ∶ X N ↦ R is continuous on the compact domain X N , it is also uniformly continuous and, for any given level of tolerance ϵ > 0 , there is a ∆ x such that for any x ′ , x ′′ ∈ X N with distance ∥ x ′ -x ′′ ∥ 2 ≤ ∆ x we have ∣ f ( x ′ ) -f ( x ′′ )∣ ≤ ϵ . For a given x , the distance between any ξ ∈ C x and any ω ∈ Ω x is ∥ ξ -ω ∥ 2 ≤ δ √ N . Since δ = 1 / n we can select n so that

6.3 Universal approximation properties with linear output neuron

In this case, the approximation capabilities of our network over the whole domain depend on the local behaviour of subnetworks converging not in a single second-hidden-layer neuron but in 2 N second-hidden-layer neurons.

Formally, given a center ω ∈ X N we include in a subnetwork neurons of the second hidden layer with outputs labelled z ω 1 -, z ω 1 + ,. . . , z ω N -, z ω N + as well as all the previous neurons needed to compute such outputs.

The expression of each z ω j ± is given by Lemma 1 and thus is defined by the center point ω j ± = ( ω j ± 1 , . . . , ω j ± N ) , by the slopes δ L ,j ± 1 , . . . , δ L ,j ± N and δ R ,j ± 1 , . . . , δ R ,j ± N , as well as by the side lengths l j ± 1 , . . . , l j ± N .

In a subnetwork, everything depends on two quantities δ, ℓ ≥ 0 that are used to set

x

1

x

1

x

1

ω

1

-

ω

2

-

ω

2+

ω

ω

ω

ω

1+

x

1

z

ω

1

-

(

x

1

, x

2

)

z

ω

1+

(

x

1

, x

2

)

z

ω

2

-

(

x

1

, x

2

)

z

ω

2+

(

x

1

, x

2

)

x

2

x

2

x

2

x

2

ω

1

-

ω

ω

1+

ω

2+

ω

2

-

1.0

0.5

1.0

00

0.0

0.5

0.5

1.0

00A

1.0

0.5

1.0

0.0

0.0

0.5

0.5

1.0

0'0

1.0

0.5

1.0

0.0

0.0

0.5

0.5

1.0

0'0

1.0

0.5

1.0

0.0

0.0

70.5

0.5

1 ∩

20.0

1

The approximation capabilities depend on the behaviour of the output of the subnetworks in the three 176 disjoint domains X ∎ ω , X /uni25FB ω , and X N ∖ X ω . 177

It is easy to see that if x ∈ X N ∖ X ω then z ω j ± = 0 for j = 1 , . . . , N . 178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

For x ∈ X ∎ ω the following Lemma holds.

Lemma 2. Given any choice of N + 1 coefficients a and b j for j = 1 , . . . , N , one may choose 2 N weights c j ± with j = 1 , . . . , N such that

for any x ∈ X ∎ ω , where Z ω ( x ) remains implicitly defined.

Proof of Lemma 2. Due to the definition of ω j ± we have

Hence, if x ∈ X ∎ ω we know that ω j -j ≤ x j ≤ ω j + j for j = 1 , . . . , N .

Moreover, since by definition for any i, j = 1 , . . . , N and i ≠ j we have ω j + i -ω j -i = 2 ℓ and ω j -i + ω j + i = 2 ω i , then ∣ x i -ω j ± i ∣ ≤ ℓ when i ≠ j . Therefore, one can apply Lemma 1 and compute ∆ ( x ) , for which all the terms in (6) but ∣ x j -ω j ± j ∣ are non-positive, thus yielding z ω j ± ( x ) = 1 -∣ x j -ω j ± j ∣ / ( 2 ℓ ) .

Without any loss of generality, translate X ω so that ω = ( ℓ, . . . , ℓ ) . This implies ω j -j = 0 and ω j + j = 2 ℓ for j = 1 , . . . , N , thus yielding z ω j -( x ) = 1 -x j 2 ℓ and z ω j + ( x ) = x j 2 ℓ . With this,

that can yield any affine function f ( x ) = a + ∑ N j = 1 b j x j by setting, for j = 1 , . . . , N ,

Finally, what happens for x ∈ X /uni25FB ω is described by the following Lemma.

Lemma 3. If the 2 N weights c j ± with j = 1 , . . . , N are set according to Lemma 2 so that Z ω ( x ) = a +∑ N j = 1 b j x j for any x ∈ X ∎ ω , for coefficients satisfying ∣ a ∣ , ∣ b j ∣ ≤ M for some M > 0 and j = 1 , . . . , N , then ∣ Z ω ( x )∣ ≤ 3 MN for any x ∈ X ω and thus for any x ∈ X /uni25FB ω .

Proof of Lemma 3. From ∣ a ∣ , ∣ b j ∣ ≤ M and from (12) we get ∣ c j -∣ ≤ M / N and ∣ c j + ∣ ≤ M / N + 2 ℓM . 196

Overall, since ℓ ≤ 1 and N ≥ 1 we have ∣ c j ± ∣ ≤ 3 M . Since 0 ≤ z ω j ± ≤ 1 and Z ω ( x ) = 197 ∑ N j = 1 c j ± z ω j ± ( x ) we finally get the thesis. 198

199

200

The above characterization of the output of Z -subnetworks allows to prove their local approximation capabilities.

201

202

Lemma 4. Given any function f ∈ C 2 ( X N ) , there are two constants P,Q > 0 such that

with o (⋅) = 1 -1 / ( 1 +⋅) N

Proof of Lemma 4. Since f ∈ C 2 ( X N ) and X N is compact, M 0 , M 1 , M 2 ≥ 0 exists such that 203

for any x ∈ X M and i, j = 1 , . . . , N .

204

Assuming x ∈ X ∎ ω , and thus ∣ x i -ω i ∣ ≤ ℓ , the above bounds can be used jointly with the Taylor 205 expansions of f and its derivatives around ω 206

to ensure that their error terms satisfy 207

and 208

Again focusing on x ∈ X ∎ ω , exploit Lemma 2 to set the weights c j ± yielding 209

which is also such that ∂Z ω ∂x i ( x ) = ∂f ∂x u ( ω ) . 210

Hence, we may program Z ω to reproduce the beaviour of f and its derivatives in X ∎ ω , and the 211 approximation errors can be derived exploiting (14) with (16) and (15) with (17) to obtain 212

To address the case x ∈ X /uni25FB ω , we may apply Lemma 3. By matching (18) with (13) we get that 213 ∣ a ∣ ≤ M 0 + M 1 N and ∣ b i ∣ ≤ M 1 ≤ M 0 + M 1 N for i = 1 , . . . , N . Hence, if x ∈ X /uni25FB ¯ ω , then if 214 M 3 = M 0 ( 1 + 3 N ) + 3 M 1 N 2 we have 215

Since we have different error bounds in X ∎ ω and X /uni25FB ¯ ω , we bound the overall error E ω by splitting 216

and apply (19) and (20) to bound each integrand. Adding the fact that the measure of X ∎ ω is ( 2 ℓ ) N , 217 while the measure of X /uni25FB ω is ( 2 ℓ + 2 δ ) N -( 2 ℓ ) N we obtain 218

219

220

221

222

223

224

from which we may set P = ( 1 2 M 2 N 2 ) p +( 1 2 M 2 N ) p and Q = M p 3 +( 2 M 1 ) p to get the thesis.

We are now in the position of proving our second result.

Proof of Theorem 2. For n > 0 integer define δ and ℓ such that δ = ℓ 2 and 2 ℓ + 2 δ = 1 / n . Let also Ω = { 1 2 n , 3 2 n , . . . , 2 n -1 2 n } N so that X N is partitioned in n N hyper-cubes X ω with centers ω ∈ Ω and side 2 ℓ + 2 δ . The output of the whole network is Z ( x ) = ∑ ω ∈ Ω Z ω ( x ) .

Since Z ω ( x ) is null for x / ∈ X ω , the error measure over X N can be decomposed into

Each of the terms in the last sum can be bounded using Lemma 4 in which we may also substitute 225 2 ℓ + 2 δ = 1 / n and δ = ℓ 2 to yield 226

227

228

229

230

231

232

233

234

235

236

Since when n →∞ we have ℓ → 0 and thus o ( ℓ ) → 0 the thesis is proven.

7 Conclusions

We established that neural networks in which hidden MAC neurons are substituted with MAM neurons to obtain more aggressively prunable architectures are still universal approximators.

References

L. Prono, P. Bich, M. Mangia, F. Pareschi, R. Rovatti, and G. Setti, 'A Multiply-And-Max/min Neuron Paradigm for Aggressively Prunable Deep Neural Networks,' Apr. 2023, preprint available on TechRxiv.

P. Bich, L. Prono, M. Mangia, F. Pareschi, R. Rovatti, and G. Setti, 'Aggressively prunable MAM²-based Deep Neural Oracle for ECG acquisition by Compressed Sensing,' in 2022 IEEE Biomedical Circuits and Systems Conference (BioCAS) , Oct. 2022, pp. 163-167.

G. Cybenko, 'Approximation by superpositions of a sigmoidal function,' Mathematics of Control, Signals 237 and Systems , vol. 2, no. 4, pp. 303-314, Dec. 1989. 238

L.-X. Wang, 'Fuzzy systems are universal approximators,' in [1992 Proceedings] IEEE International 239 Conference on Fuzzy Systems , 1992, pp. 1163-1170. 240

B. Kosko, 'Fuzzy systems as universal approximators,' IEEE Transactions on Computers , vol. 43, no. 11, 241 pp. 1329-1333, 1994. 242

J. Castro, 'Fuzzy logic controllers are universal approximators,' IEEE Transactions on Systems, Man, and 243 Cybernetics , vol. 25, no. 4, pp. 629-635, Apr. 1995. 244

R. Rovatti, 'Fuzzy piecewise multilinear and piecewise linear systems as universal approximators in 245 sobolev norms,' IEEE Transactions on Fuzzy Systems , vol. 6, no. 2, pp. 235-249, 1998. 246

K. Hornik, M. Stinchcombe, and H. White, 'Multilayer feedforward networks are universal approximators,' 247 Neural Networks , vol. 2, no. 5, pp. 359-366, 1989. 248

A. Pinkus, 'Approximation theory of the MLP model in neural networks,' Acta Numerica , vol. 8, pp. 249 143-195, Jan. 1999. 250

G. Gripenberg, 'Approximation by neural networks with a bounded number of nodes at each level,' Journal 251 of Approximation Theory , vol. 122, no. 2, pp. 260-266, Jun. 2003. 252

Z. Lu, H. Pu, F. Wang, Z. Hu, and L. Wang, 'The Expressive Power of Neural Networks: A View from the 253 Width,' in Advances in Neural Information Processing Systems , vol. 30. Curran Associates, Inc., 2017. 254

B. Hanin and M. Sellke, 'Approximating Continuous Functions by ReLU Nets of Minimal Width,' Mar. 255 2018. 256

V. Maiorov and A. Pinkus, 'Lower bounds for approximation by MLP neural networks,' Neurocomputing , 257 vol. 25, no. 1, pp. 81-91, Apr. 1999. 258

N. J. Guliyev and V. E. Ismailov, 'On the approximation by single hidden layer feedforward neural 259 networks with fixed weights,' Neural Networks , vol. 98, pp. 296-304, Feb. 2018. 260

Y. Cai, 'Achieve the Minimum Width of Neural Networks for Universal Approximation,' in The Eleventh 261 International Conference on Learning Representations , Feb. 2023. 262

D.-X. Zhou, 'Universality of deep convolutional neural networks,' Applied and Computational Harmonic 263 Analysis , vol. 48, no. 2, pp. 787-794, 2020. 264

O. A. Manita, M. A. Peletier, J. W. Portegies, J. Sanders, and A. Senen-Cerda, 'Universal approximation 265 in dropout neural networks,' J. Mach. Learn. Res. , vol. 23, no. 1, jan 2022. 266

Y. Lu and J. Lu, 'A universal approximation theorem of deep neural networks for expressing probability 267 distributions,' in Advances in Neural Information Processing Systems , vol. 33. Curran Associates, Inc., 268 2020, pp. 3094-3105. 269

S.-Q. Zhang and Z.-H. Zhou, 'Theoretically provable spiking neural networks,' in Advances in Neural 270 Information Processing Systems , vol. 35. Curran Associates, Inc., 2022, pp. 19 345-19 356. 271

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript proves universal approximation results for MAM neurons. MAM neurons are essentially ReLU neurons that operate on the sum of the maximum and the minimum of the weighted inputs, plus a bias. Previous work claims these neurons are useful for reducing the memory footprint of deep neural networks. Thus, in short, the paper proves how any real-valued continuous function defined on a compact set can be approximated to any arbitrary degree of precision by a network consisting primarily of MAM neurons (for instance in the sup norm).\n\nSTRENGTHS:\nThe paper is relatively well-written and easy to understand. Universal approximation properties are important, and proving them is a key step when new activation functions are introduced.\n\nWEAKNESSES:\nThere are two problems with this contribution. First, there are many universal approximation results in the literature already, and thus this contribution is perceived as incremental. Second, and more importantly, an even better result can easily be proved by adapting well-known techniques for providing simple proofs of universal approximation properties (the authors do not seem to be aware of the existence of such techniques).One only needs to slightly tweak the proof that is used for the case of ReLU neurons.   In particular, it is easy to show that any continuous function from [0,1] to R, can be approximated by a neural network with a single linear output unit and two hidden layers of MAM neurons to any degree of precision \\epsilon. To see this, note that f is *uniformly* continuous over [0,1]. Thus the [0,1] interval can be subdivided into small intervals of size \\alpha, such that within any such interval f is contained \"within a sleeve of thickness \\epsilon\". Now connect the input to all the MAM neurons in the first hidden layer with identical weights equal to 0.5. As a result, the max + min portion of the input of each MAM neuron in the first hidden layer is equal to the input x. Now select a sequence of (arithmetically) increasing biases so that:  \n1) the first hidden neuron is turned on if x in the first interval of size \\alpha and all the other hidden neurons produce a zero;\n2) the first and second neurons are turned on if x is in the second interval of size \\alpha and all other hidden neurons produce a zero;\netc. In other words, essentially code the value of x by the number of neurons that are turned on in the hidden layer. It is then easy to see how to design the following hidden layer and connect it to the single linear output neuron to obtain the desired approximation.\n\nQUESTIONS:\nThe authors may consider providing a simpler proof, along the lines sketched above, for a potential submission to a lesser conference.\n\nLIMITATIONS:\nThe main limitation of the universal approximation results is that the hidden layers can be arbitrary large (depending on the size of \\alpha in the proof sketched above). And thus in general the constructive proofs of these results are not practical. This is a well known limitation and the authors, to their credit, do mention it.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper demonstrates that the network can still maintain the universal approximation property after substituting the classical MAC hidden neurons of neural networks with the MAM neurons, which only rely on the maximum and minimum elements of the summation, allowing for more aggressive pruning. Specifically, the authors consider a network with two hidden MAM layers and two kinds of output layers. They show that the networks can achieve universal approximation capabilities under different norms for target functions with varying smoothness. The constructive proof of the first case utilizes a similar idea to the partition of unity, and the second one decomposes the whole domain and deals with the local behavior of the subnetworks involving the second hidden layer.\n\nSTRENGTHS:\nThe authors provide a universal approximation result for a recently proposed hidden neuron for neural networks with two different output designs. The constructive proof introduces localized hyper-rectangles, which may inspire other step-function-like construction in some approximation problems. It’s also helpful to provide some intuitive illustrations of the construction.\n\nWEAKNESSES:\nWhile the result is somewhat interesting, it fails to provide more compelling evidence for the newly proposed MAM neurons. In the introduction, MAM neurons’ main advantage is that they can allow more aggressive pruning. But the theorems and the proof process seem to have no investigation about these properties. And there is no further discussion about the academic and practical potentials of MAM neurons, weakening its significance and attractiveness as well as this work.\n\nIn Section 5, the authors claim that the theorem has no constraints on the layer width, which contradicts the conventional universal approximation properties. However, the proof introduces a parameter n that needs to be chosen sufficiently large (See line 158). Interestingly, this parameter seems to be related to the width or scale of the neural network according to its definition in line 141. Hence, there appears to be a potential contradiction in this context.\n\nThere are several evident traces of incompleteness throughout the manuscript, such as lines 141-145, equation after line 170 and line 216, and the unfinished Conclusions section in lines 229-230, indicating that this paper was hastily written and has not undergone thorough revisions.\n\nThe inadequate mathematical formatting in the manuscript has resulted in difficulties in comprehending the proof. Some mathematical notations used in this paper contradict commonly used notations in classical theories. The LaTeX formatting in the manuscript is not standardized.\n\nQUESTIONS:\nOne critical issue with the inadequate mathematical formatting is that there is a lack of notation related to the width of the hidden layers in the network, resulting in significant ambiguity regarding the lengths of vectors and the ranges of summation throughout the manuscript, especially in the proof of Lemma 1. \n\nSome mathematical notations may need to be changed, such as the norm defined after line 81, which might be easily misread as the Sobolev norm, the L’(x) and L’’(x) in line 74 are confusing and could be mistaken for the derivative of the function L(x), and o( ) in line 202 contradicts the standard infinitesimal notation.\n\nThere are instances where symbols are misused, such as \\setminus in line 177, Eq. (6), fractional form in line 187, and o( ) in line 202, and unnecessary empty lines precede certain equations, just like Eq. 7, 9, 14, 16, 17.\n\nThe authors give two theorems for different settings, each with various norms, smoothness, and output designs. However, the relationship between these two theorems is not clear. It would be better to show that one is a corollary of the other. \n\nBesides, I hope the authors could find more considerable advantages of this unique neuron through the theoretical investigation, which would strengthen the paper’s contributions.\n\nLIMITATIONS:\nThe authors do discuss the limitations in Section 5. However, due to the issue with the parameter n (See Weaknesses), I think the discussion is insufficient. Besides, they mention that the theoretical results wouldn’t directly lead to an efficient approximation, so I wonder if there are any numerical experiments conducted based on the setting in this paper. Besides, how large is the gap between the theoretical setting and practical applications since this paper has many assumptions and hypotheses?", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents two universal approximation theorems for deep neural networks associated with a so-called Multiply-And-Max/min (MAM) activation function defined with the maximum and minimum of the input components and a bias constant. One is for uniform approximation and the other for approximation in the Sobolev space W^{1, p} with p\\ge 1.\n\nSTRENGTHS:\nThe presented universal approximation theorems are interesting and should have implications to show the power of pooling layers in deep neural networks. Studying simultaneous approximation in terms of the norm in the Sobolev space W^{1, p} should be able to explain the efficiency of some deep learning algorithms. These ideas are novel in my opinion.\n\nWEAKNESSES:\nThough the approximation theorems are novel to me, the paper has a few weak points and should be improved:\n\n1. To demonstrate some theoretical advantages of the MAM activation function. This might be done with the pooling layers in deep neural networks. \n\n2. To present rates of approximation. Quantitative estimates for approximation of functions in various function spaces are crucial in the approximation error estimate for generalization analysis of deep learning algorithms. \n\n3. To give rigorous statements and proofs. In Lemma 1, the sentence \"Let z be any ... layer.\" should be removed because the output z is constructed by (5) and is not an arbitrary output function. In its proof, \"We assume ...\" and \"the output ... only one of the inputs\" should be revised: the neurons are constructed by (7), not by assumption. In Lemma 4, P is not a constant. It is a quantity depending on \\ell of the form constant + o(1/\\ell).  \n\n4. To give fair credits to the existing literature. For example, the construction of trapezoid functions has a long history in the study of deep neural networks and can be found in the papers of Shaham-Cloninger-Coifman (2018), Chui-Lin-Zhang-Zhou (2020), and some others.\n\nQUESTIONS:\nSome questions listed in the previous section should be answered.\n\nLIMITATIONS:\nBetter theoretical results would improve the quality of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the universal approximation properties of ReLU networks using the Multiply-And-Max/min (MAM) neurons. Literature on the universal approximation properties of ReLU networks using the Multiply-and-ACcumulate (MAC) neurons is vast. However, the study on MAM neurons seems lacking. Hence, two theorems taking a step in characterizing the universal approximation properties for MAM neurons are proved in this paper. The first theorem states that a two-hidden-layer ReLU network using MAM neurons in the first two layers and the normalized linear combination in the last layer can approximate any continuous function on a unit hypercube arbitrarily well in terms of the infinity norm. The second theorem is similar to the first one, stating that a two-hidden-layer ReLU network using MAM neurons in the first two layers and the linear combination in the last layer can approximate any twice continuously differentiable function on a unit hypercube arbitrarily well in terms of the Sobolev norm. The proofs of these two theorems are constructive and the authors also acknowledge that their results do not imply efficient approximation.\n\nSTRENGTHS:\nThe novelty is clear. The novel contribution of this paper is apparently the theoretical guarantees on the universal approximation properties of ReLU networks using MAM neurons. Although I have not carefully validated the proof, the explanations and statements given in the paper seem to be sufficiently convincing. Overall, this is a well-written paper. The presentation is concise, clear, and easy to follow. I enjoy reading the paper.\n\nWEAKNESSES:\n1. The requirement of the target function being twice continuously differentiable seems a bit limited. It would be great if the authors could relax this assumption or clarify why this assumption is necessary.\n2. The result in Theorem 2 relies on the L^p Sobolev norm. Would it be possible to extend the result to the infinity norm? The paper would be more convincing and clearer if the authors can justify why the normalized linear combination and the linear combination use different norms. The connection between Theorem 1 and 2 seems missing.\n3. Given the observation that using MAM or the mixed MAM/MAC neurons gives better pruning performance than the MAC neurons in practice, the paper would be more convincing if the authors can provide some insights into the constructive approximation of these different schemes.\n\nQUESTIONS:\nLine 81-82: Is the second term a partial derivative of phi with respect to x_j? There is a missing partial symbol in the denominator.\n\nLine 112: The authors mention “...weakly unimodal piecewise-linear functions…” What does the word “weakly” mean here? Can we use unimodal piecewise-linear functions?\n\nThe two theorems provided in the paper rely on the assumption of using a two-hidden-layer network. I think it may be trivial to extend the results to multiple layers. Can we extend the results to a single-hidden-layer network? If not, would it be possible to prove it? It would be nice to give some insights into this.\n\nLIMITATIONS:\nThe authors clearly state the limitations of their work in Section 5. Specifically, they point out that their results do not imply efficient implementation. I’m glad to see they make it very clear. They also state that the efficiency of approximation will be their future focus. I think this paper has laid a good foundation for their future work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}