Title: Computational Complexity of Detecting Proximity to Losslessly Compressible Neural Network Parameters
Venue: NeurIPS 2023
Accepted: no
Keywords: theory

neural network theory

structural redundancy

compressibility

lossless compressibility

computational complexity

NP-completeness
Abstract: To better understand complexity in neural networks, we theoretically investigate the idealised phenomenon of lossless network compressibility, whereby an identical function can be implemented with a smaller network. We give an efficient formal algorithm for optimal lossless compression in the setting of single-hidden-layer hyperbolic tangent networks. To measure lossless compressibility, we define the rank of a parameter as the minimum number of hidden units required to implement the same function. Losslessly compressible parameters are atypical, but their existence has implications for nearby parameters. We define the proximate rank of a parameter as the rank of the most compressible parameter within a small $L^\infty$ neighbourhood. Unfortunately, detecting nearby losslessly compressible parameters is not so easy: we show that bounding the proximate rank is an NP-complete problem, using a reduction from Boolean satisfiability via a novel abstract clustering problem involving covering points with small squares. These results underscore the computational complexity of measuring neural network complexity, laying a foundation for future theoretical and empirical work in this direction.


Abstract

To better understand complexity in neural networks, we theoretically investigate the idealised phenomenon of lossless network compressibility, whereby an identical function can be implemented with a smaller network. We give an efficient formal algorithm for optimal lossless compression in the setting of single-hiddenlayer hyperbolic tangent networks. To measure lossless compressibility, we define the rank of a parameter as the minimum number of hidden units required to implement the same function. Losslessly compressible parameters are atypical, but their existence has implications for nearby parameters. We define the proximate rank of a parameter as the rank of the most compressible parameter within a small L ∞ neighbourhood. Unfortunately, detecting nearby losslessly compressible parameters is not so easy: we show that bounding the proximate rank is an NP -complete problem, using a reduction from Boolean satisfiability via a novel abstract clustering problem involving covering points with small squares. These results underscore the computational complexity of measuring neural network complexity, laying a foundation for future theoretical and empirical work in this direction.

1 Introduction

Learned neural networks often generalise well, depite the excessive expressive capacity of their ar17 chitectures (Zhang et al., 2017, 2021). Moreover, learned neural networks are often approximately 18 compressible , in that smaller networks can be found implementing similar functions (via, e.g., model 19 distillation, Buciluˇ a et al., 2006; Hinton et al., 2014; see, e.g., Sanh et al., 2019 for a large-scale ex20 ample). In other words, learned neural networks are often simpler than they might seem. Advancing 21 our understanding of neural network complexity is key to understanding deep learning. 22

We propose studying the idealised phenomenon of lossless compressibility , whereby an identical 23

imply that, in many architectures, almost all parameters are incompressible in this lossless, unit25

function can be implemented with a smaller network. 1 Classical functional equivalence results 24

based sense (e.g., Sussmann, 1992; Chen et al., 1993; Fefferman, 1994; Phuong and Lampert, 2020). 26

functional equivalence classes (Anonymous, 2023), some of which are losslessly compressible. 28

However, these results specifically exclude measure zero sets of parameters with more complex 27

We argue that, despite their atypicality, losslessly compressible parameters may be highly relevant

29 to deep learning. The learning process exerts a non-random selection pressure on parameters, and 30 losslessly compressible parameters are appealing solutions due to parsimony. Moreover, losslessly 31 compressible parameters are a source of information singularities (cf. Fukumizu, 1996), highly rele32 vant to statistical theories of deep learning (Watanabe, 2009; Wei et al., 2022). 33

Even if losslessly compressible parameters themselves are rare, their aggregate parametric neigh34 bourhoods have nonzero measure. These neighbourhoods have a rich structure that reaches through35 out the parameter space (Anonymous, 2023). The parameters in these neighbourhoods implement 36 similar functions to their losslessly compressible neighbours, so they are necessarily approximately 37 compressible. Their proximity to information singularities also has implications for local learning 38 dynamics (Amari et al., 2006; Wei et al., 2008; Cousseau et al., 2008; Amari et al., 2018). 39

40

41

42

43

44

In this paper, we study losslessly compressible parameters and their neighbours in the setting of single-hidden-layer hyperbolic tangent networks. While this architecture is not immediately relevant to modern deep learning, parts of the theory are generic to feed-forward architecture components. A comprehensive investigation of this simple and concrete case is a first step towards studying more modern architectures. To this end, we offer the following theoretical contributions.

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

In Section 4, we give efficient formal algorithms for optimal lossless compression of singlehidden-layer hyperbolic tangent networks, and for computing the rank of a parameters-the minimum number of hidden units required to implement the same function.

In Section 5, we define the proximate rank -the rank of the most compressible parameter within a small L ∞ neighbourhood. We give a greedy algorithm for bounding this value.

In Section 6, we show that bounding the proximate rank below a given value (that is, detecting proximity to parameters with a given maximum rank), is an NP -complete decision problem. The proof involves a reduction from Boolean satisfiability via a novel abstract decision problem involving clustering points in the plane into small squares.

These results underscore the computational complexity of measuring neural network complexity: we show that while lossless network compression is easy, detecting highly-compressible networks near a given parameter can be very hard indeed (embedding any computational problem in NP ). Our contributions lay a foundation for future theoretical and empirical work detecting proximity to losslessly compressible parameters in learned networks using modern architectures. In Section 7, we discuss these research directions, and limitations of the lossless compressibility framework.

2 Related work 2

Two neural network parameters are functionally equivalent if they implement the same function. In single-hidden-layer hyperbolic tangent networks, Sussmann (1992) showed that, for almost all parameters, two parameters are functionally equivalent if and only if they are related by simple operations of exchanging and negating the weights of hidden units. Similar operations have been found for various architectures, including different nonlinearities (e.g., Albertini et al., 1993; K˚ urková and Kainen, 1994), multiple hidden layers (e.g., Fefferman and Markel, 1993; Fefferman, 1994; Phuong and Lampert, 2020), and more complex connection graphs (Vlaˇ ci´ c and Bölcskei, 2021, 2022).

Lossless compressibility requires functionally equivalent parameters in smaller architectures. In all architectures where functional equivalence has been studied (cf. above), the simple operations identified do not change the number of units. However, all of these studies explicitly exclude from consideration certain measure zero subsets of parameters with richer functional equivalence classes. The clearest example of this crucial assumption comes from Sussmann (1992), whose result holds exactly for 'minimal networks' (in our parlance, losslessly incompressible networks).

Anonymous (2023) relaxes this assumption, studying functional equivalence for non-minimal singlehidden-layer hyperbolic tangent networks. Anonymous (2023) gives an algorithm for finding canonical equivalent parameters using various opportunities for eliminating or merging redundant units. 3 This algorithm implements optimal lossless compression as a side-effect. We give a more direct and efficient lossless compression algorithm using similar techniques.

Beyond lossless compression, there is a significant empirical literature on approximate compressibility and compression techniques in neural networks, including via network pruning, weight quantisation, and student-teacher learning (or model distillation). Approximate compressibility has also

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

been proposed as a learning objective (see, e.g., Hinton and van Camp, 1993; Aytekin et al., 2019) and used as a basis for generalisation bounds (Suzuki et al., 2020a,b). For an overview, see Cheng et al. (2018, 2020) or Choudhary et al. (2020). Of particular interest is a recent empirical study of network pruning from Casper et al. (2021), who, while investigating the structure of learned neural networks, found many instances of units with weak or correlated outputs. Casper et al. (2021) found that these units could be removed without a large effect on performance, using elimination and merging operations bearing a striking resemblance to those discussed by Anonymous (2023).

3 Preliminaries

We consider a family of fully-connected, feed-forward neural network architectures with one input unit, one biased output unit, and one hidden layer of h ∈ N biased hidden units with the hyperbolic tangent nonlinearity tanh( z ) = ( e z -e -z ) / ( e z + e -z ) . The weights and biases of the network are encoded in a parameter vector in the format w = ( a 1 , b 1 , c 1 , . . . , a h , b h , c h , d ) ∈ W h = R 3 h +1 , where for each hidden unit i = 1 , . . . , h there is an outgoing weight a i ∈ R , an incoming weight b i ∈ R , and a bias c i ∈ R ; and d ∈ R is the output unit bias . Thus each parameter w ∈ W h indexes a mathematical function f w : R → R such that f w ( x ) = d + ∑ h i =1 a i tanh( b i x + c i ) . All of our results generalise to networks with multi-dimensional inputs and outputs.

Two parameters w ∈ W h , w ′ ∈ W h ′ are functionally equivalent if f w = f w ′ as functions on R ( ∀ x ∈ R , f w ( x ) = f w ′ ( x ) ). A parameter w ∈ W h is (losslessly) compressible (or non-minimal ) if and only if w is functionally equivalent to some w ′ ∈ W h ′ with fewer hidden units h ′ < h (otherwise, w is incompressible or minimal ). Sussmann (1992) showed that a simple condition, reducibility, is necessary and sufficient for lossless compressibility. A parameter ( a 1 , b 1 , c 1 , . . . , a h , b h , c h , d ) ∈ W h is reducible if and only if it satisfies any of the following reducibility conditions :

/negationslash

/negationslash

Each reducibility condition suggests a simple operation to remove a hidden unit while preserving the function (Sussmann, 1992; Anonymous, 2023): (i) units with zero outgoing weight do not contribute to the function; (ii) units with zero incoming weight contribute a constant that can be incorporated into the output bias; and (iii), (iv) unit pairs with identical (negative) incoming weight and bias contribute in proportion (since the hyperbolic tangent is odd), and can be merged into a single unit with the sum (difference) of their outgoing weights.

Define the uniform norm (or L ∞ norm ) of a vector v ∈ R p as ‖ v ‖ ∞ = max p i =1 abs( v i ) , the largest absolute component of v . Define the uniform distance between v and u ∈ R p as ‖ u -v ‖ ∞ . Given a positive scalar ε ∈ R + , define the closed uniform neighbourhood of v with radius ε , ¯ B ∞ ( v ; ε ) , as the set of vectors of distance at most ε from v : ¯ B ∞ ( v ; ε ) = { u ∈ R p : ‖ u -v ‖ ∞ ≤ ε } .

A decision problem 4 is a tuple ( I, J ) where I is a set of instances and J ⊆ I is a subset of affirmative instances . A solution is a deterministic algorithm that determines if any given instance i ∈ I is affirmative ( i ∈ J ). A reduction from one decision problem X = ( I, J ) to another Y = ( I ′ , J ′ ) is a deterministic polytime algorithm implementing a mapping ϕ : I → I ′ such that ϕ ( i ) ∈ J ′ ⇔ i ∈ J . If such a reduction exists, say X is reducible 5 to Y and write X → Y . Reducibility is transitive.

P is the class of decision problems with polytime solutions (polynomial in the instance size). NP is the class of decision problems for which a deterministic polytime algorithm can verify affirmative instances given a certificate. A decision problem Y is NP -hard if all problems in NP are reducible to Y ( ∀ X ∈ NP , X → Y ). Y is NP -complete if Y ∈ NP and Y is NP -hard. Boolean satisfiability is a well-known NP -complete decision problem (Cook, 1971; Levin, 1973; see also Garey and Johnson, 1979). NP -complete decision problems have no known polytime exact solutions.

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

4 Lossless compression and rank

We consider the problem of lossless neural network compression: finding, given a compressible parameter, a functionally equivalent but incompressible parameter. The following algorithm solves this problem by eliminating units meeting reducibility conditions (i) and (ii), and merging unit pairs meeting reducibility conditions (iii) and (iv) in ways preserving functional equivalence.

Algorithm 4.1 (Lossless neural network compression) . Given h ∈ N , proceed:

/negationslash

/negationslash

Theorem 4.1 (Algorithm 4.1 correctness) . Given w ∈ W h , compute w ′ = COMPRESS ( w ) ∈ W r . (i) f w ′ = f w , and (ii) w ′ is incompressible.

Proof sketch (Full proof in Appendix A) . For (i), note that units eliminated in Stage 1 contribute a constant a i tanh( c i ) , units merged in Stage 2 have proportional contributions ( tanh is odd), and merged units eliminated in Stage 3 do not contribute. For (ii), by construction, w ′ satisfies no reducibility conditions, so w ′ is not reducible and thus incompressible by Sussmann (1992). ♦

Wedefine the rank 6 of a neural network parameter w ∈ W h , denoted rank( w ) , as the minimum number of hidden units required to implement f w : rank( w ) = min { h ′ ∈ N : ∃ w ′ ∈ W h ′ ; f w = f w ′ } . The rank is also the number of hidden units in COMPRESS ( w ) , since Algorithm 4.1 produces an incompressible parameter, which is minimal by definition. Computing the rank is therefore a trivial matter of counting the units, after performing lossless compression. The following is a streamlined algorithm, following Algorithm 4.1 but removing steps that don't influence the final count.

Algorithm 4.2 (Rank of a neural network parameter) . Given h ∈ N , proceed:

/negationslash

/negationslash

Theorem 4.2 (Algorithm 4.2 correctness) . Given w ∈ W h , rank( w ) = RANK ( w ) .

Proof. Let r be the number of hidden units in COMPRESS ( w ) . Then r = rank( w ) by Theorem 4.1. Moreover, comparing Algorithms 4.1 and 4.2, observe RANK ( w ) = r .

Remark 4.3. Both Algorithms 4.1 and 4.2 require O ( h log h ) time if the partitioning step is performed by first sorting the units by lexicographically non-decreasing sign( b i ) · ( b i , c i ) .

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

5 Proximity to low-rank parameters

Given a neural network parameter w ∈ W h and a positive radius ε ∈ R + , we define the proximate rank of w at radius ε , denoted prank ε ( w ) , as the rank of the lowest-rank parameter within a closed uniform ( L ∞ ) neighbourhood of w with radius ε . That is,

The proximate rank measures the proximity of w to the set of parameters with a given rank bound, that is, sufficiently losslessly compressible parameters.

The following greedy algorithm computes an upper bound on the proximate rank. The algorithm replaces each of the three stages of Algorithm 4.2 with a relaxed version, as follows.

Instead of eliminating units with zero incoming weight, eliminate units with near zero incoming weight (there is a nearby parameter where these are zero).

Instead of partitioning the remaining units by sign( b i ) · ( b i , c i ) , cluster them by nearby sign( b i ) · ( b i , c i ) (there is a nearby parameter where they have the same sign( b i ) · ( b i , c i ) ).

Instead of eliminating merged units with zero outgoing weight, eliminate merged units with near zero outgoing weight (there is a nearby parameter where these are zero).

Step (2) is non-trivial, we use a greedy approach, described separately as Algorithm 5.2.

214

Theorem 5.1 (Algorithm 5.1 correctness) . For w ∈ W h and ε ∈ R + , prank ε ( w ) ≤ BOUND ( ε, w ) .

215

216

217

218

219

220

221

222

223

224

Proof sketch (Full proof in Appendix A) . Trace the algorithm to construct a parameter u ∈ ¯ B ∞ ( w ; ε ) with rank( u ) = BOUND ( ε, w ) . During Stage 1, set the nearly-eliminable incoming weights to zero. Use the group-starting vectors v 1 , . . . , v J from Algorithm 5.2 to construct mergeable incoming weights and biases during Stage 2. During Stage 3, subtract or add a fraction of the merged unit outgoing weight from the outgoing weights of the original units. ♦

Remark 5.2. Both Algorithms 5.1 and 5.2 have worst-case runtime complexity O ( h 2 ) .

Remark 5.3. Algorithm 5.1 does not compute the proximate rank-merely an upper bound. There may exist a more efficient approximate partition than the one found by Algorithm 5.2. It turns out that this suboptimality is fundamental-computing a smallest approximate partition is NP -hard, and can be reduced to computing the proximate rank. We formally prove this observation below.

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

6 Computational complexity of proximate rank

Remark 5.3 alludes to an essential difficulty in computing the proximate rank: grouping units with similar (up to sign) incoming weight and bias pairs for merging. The following abstract decision problem, Problem UPC , captures the related task of clustering points in the plane into groups with a fixed maximum uniform radius. 7

Given h source points x 1 , . . . , x h ∈ R 2 , define an ( r, ε ) -cover , a collection of r covering points y 1 , . . . , y r ∈ R 2 such that the uniform distance between each source point and its nearest covering point is at most ε (that is, ∀ i ∈ { 1 , . . . , h } , ∃ j ∈ { 1 , . . . , r } , ‖ x i -y j ‖ ∞ ≤ ε ).

Problem UPC . Uniform point cover, or UPC , is a decision problem. The instances are tuples of the form ( h, r, ε, X ) where h, r ∈ N ; ε ∈ R + ; and X is a list of h source points in R 2 . The affirmative instances are all tuples ( h, r, ε, X ) for which there exists an ( r, ε ) -cover of the h points in X .

Theorem 6.1. Problem UPC is NP -complete.

Proof sketch (Full proof in Appendix C) . The main task is to show that UPC is NP -hard ( ∀ X ∈ NP , X → UPC ). Since reducibility is transitive, it suffices to give a reduction from the well-known NP -complete problem Boolean satisfiability (Cook, 1971; Levin, 1973). Actually, to simplify the proof, we consider an NP -complete variant of Boolean satisfiability, restricted to formulas with (i) two or three literals per clause, (ii) one negative occurrence and one or two positive occurrences per literal, and (iii) a planar bipartite clause-variable incidence graph.

From such a formula we must construct a UPC instance, affirmative if and only if the formula is satisfiable. Due to the restrictions, the bipartite clause-variable is planar with maximum degree 3, and can be embedded onto an integer grid (Valiant, 1981, §IV). We divide the embedded graph into unit-width tiles of finitely many types, and we replace each tile with an arrangement of source points based on its type. The aggregate collection of source points mirrors the structure of the original formula. The variable tile arrangements can be covered essentially in either of two ways, corresponding to 'true' and 'false' in a satisfying assignment. The edge tile arrangements transfer these assignments to the clause tiles, where the cover can only be completed if all clauses have at least one true positive literal or false negative literal. Figure 1 shows one example of this construction. ♦

(a)

(b)

(c)

variables:

v

1

, . . . , v

6

clauses:

(¯

v

1

∨

¯

v

3

∨

v

4

)

(

v

1

∨

¯

v

2

∨

v

5

)

(

v

3

∨

¯

v

4

∨

v

6

)

(

v

2

∨

¯

v

5

)

(

v

5

∨

v

6

)

(

v

4

∨

¯

v

6

)

+

-

+

-

+

-

+

-

+

+

+

-

+

+

-

+

-

+

-

+

-

+

-

+

+

+

-

+

+

-

+

-

+

-

+

-

+

-

+

+

+

-

+

+

-

ε

(d)

(e)

(f)

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

The following decision problem formalises the task of bounding the proximate rank, or equivalently, detecting nearby low-rank parameters. It is NP -complete by reduction from Problem UPC .

Problem PR . Bounding proximate rank, or PR , is a decision problem. Each instance comprises a number of hidden units h ∈ N , a parameter w ∈ W h , a uniform radius ε ∈ R + , and a maximum rank r ∈ N . The affirmative instances are those instances where prank ε ( w ) ≤ r .

Theorem 6.2. Problem PR is NP -complete.

Proof. Since UPC is NP -complete (Theorem 6.1), it suffices to show UPC → PR and PR ∈ NP .

( UPC → PR , the reduction): Given an instance of Problem UPC , allocate one hidden unit per source point, and construct a parameter using the source point coordinates as incoming weights and biases. Actually, to avoid issues with zeros and signs, first translate the source points well into the positive quadrant. Likewise, set the outgoing weights to a positive value. Figure 2 gives an example.

Formally, let h, r ∈ N , ε ∈ R + , and x 1 , . . . , x h ∈ R 2 . In linear time construct a PR instance with h hidden units, uniform radius ε , maximum rank r , and parameter w ∈ W h as follows.

Define x min = ( min h i =1 x i, 1 , min h i =1 x i, 2 ) ∈ R 2 , containing the minimum first and second coordinates among all source points (minimising over each dimension independently).

Define a translation T : R 2 → R 2 such that T ( x ) = x -x min +(2 ε, 2 ε ) .

Translate the source points x 1 , . . . , x h to x ′ 1 , . . . , x ′ h where x ′ i = T ( x i ) . Note (for later) that all components of the translated source points are at least 2 ε by step (1).

Construct the neural network parameter w = (2 ε, x ′ 1 , 1 , x ′ 1 , 2 , . . . , 2 ε, x ′ h, 1 , x ′ h, 2 , 0) ∈ W h . In other words, for i = 1 , . . . , h , set a i = 2 ε , b i = x ′ i, 1 , and c i = x ′ i, 2 ; and set d = 0 .

( UPC → PR , equivalence): It remains to show that the constructed instance of PR is affirmative if and only if the given instance of UPC is affirmative, that is, there exists an ( r, ε ) -cover of the source points if and only if the constructed parameter has prank ε ( w ) ≤ r .

( ⇒ ): If there is a small cover of the source points, then the hidden units can be perturbed so that they match up with the (translated) covering points. Since there are few covering points, many units can now be merged, so the original parameter has low proximate rank.

Formally, suppose there exists an ( r, ε ) -cover y 1 , . . . , y r . Define ρ : { 1 , . . . , h } → { 1 , . . . , r } such that the nearest covering point to each source point x i is y ρ ( i ) (breaking ties arbitrarily). Then for j = 1 , . . . , r , define y ′ j = T ( y j ) where T is the translation defined in step (2) of the construction. Finally, define a parameter w /star = (2 ε, y ′ ρ (1) , 1 , y ′ ρ (1) , 2 , . . . , 2 ε, y ′ ρ ( h ) , 1 , y ′ ρ ( h ) , 2 , 0) ∈ W h (in other words, for i = 1 , . . . , h , a /star i = 2 ε , b /star i = y ′ ρ ( i ) , 1 , and c /star i = y ′ ρ ( i ) , 2 ; and d /star = 0 ).

Then rank( w /star ) ≤ r , since there are at most r distinct incoming weight and bias pairs (namely y ′ 1 , . . . , y ′ r ). Moreover, ‖ w -w /star ‖ ∞ ≤ ε , since both parameters have the same output bias and outgoing weights, and, by the defining property of the cover, for i = 1 , . . . , h ,

‖ ( b i , c i ) -( b /star i , c /star i ) ‖ ∞ = ∥ ∥ ∥ x ′ i -y ′ ρ ( i ) ∥ ∥ ∥ ∞ = ∥ ∥ T ( x i ) -T ( y ρ ( i ) ) ∥ ∥ ∞ = ∥ ∥ x i -y ρ ( i ) ∥ ∥ ∞ ≤ ε. Therefore prank ε ( w ) ≤ rank( w /star ) ≤ r .

ε

x

1

x

2

x

3

x

4

x

5

x

6

x

7

x

8

x

9

x

min

2

ε

2

ε

x

′

i

x

i

T

(

b

i

, c

i

)

b

i

2

ε

c

i

(a)

(b)

(c)

(d)

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

( ⇐ ): Conversely, since all of the weights and biases are at least 2 ε , any nearby low-rank parameter implies the approximate mergeability of some units. Therefore, if the parameter has low proximate rank, there is a small cover of the translated points, and, in turn, of the original points.

Formally, suppose prank ε ( w ) ≤ r , with w /star ∈ ¯ B ∞ ( w ; ε ) such that rank( w /star ) = r /star ≤ r . In general, the only ways that w /star could have reduced rank compared to w are the following (cf. Algorithm 4.1):

Some incoming weight b i could be perturbed to zero, allowing its unit to be eliminated.

Two units i, j with ( b i , c i ) and ( b j , c j ) within 2 ε could be perturbed to have identical incoming weight and bias, allowing them to be merged.

Two units i, j with ( b i , c i ) and -( b j , c j ) within 2 ε could be perturbed to have identically negative weight and bias, again allowing them to be merged.

Some group of m ≥ 1 units, merged through the above options, with total outgoing weight within mε of zero, could have their outgoing weights perturbed to make the total zero.

By construction, all a i , b i , c i ≥ 2 ε > 0 , immediately ruling out (1) and (3). Option (4) is also ruled out because any such total outgoing weight is 2 mε > mε . This leaves option (2) alone responsible. Thus, there are exactly r /star distinct incoming weight and bias pairs among the units of w /star . Denote these pairs y ′ 1 , . . . , y ′ r /star -they constitute an ( r /star , ε ) -cover of the incoming weight and bias vectors of w , x ′ 1 , . . . , x ′ h (as w /star ∈ ¯ B ∞ ( w ; ε ) ). Finally, invert T to produce an ( r /star , ε ) -cover of x 1 , . . . , x h , and add r -r /star arbitrary covering points to extend this to the desired ( r, ε ) -cover.

( PR ∈ NP ): We must show that an affirmative instance of PR can be verified in polynomial time, given a certificate. Consider an instance h, r ∈ N , ε ∈ R + , and w = ( a 1 , b 1 , c 1 , . . . , a h , b h , c h , d ) ∈ W h . Use as a certificate a partition 8 Π 1 , . . . , Π J of { i ∈ { 1 , . . . , h } : abs( b i ) > ε } , such that (1) for each Π j , for each i, k ∈ Π j , ‖ sign( b i ) · ( b i , c i ) -sign( b k ) · ( b k , c k ) ‖ ∞ ≤ 2 ε ; and (2) at most r of the Π j satisfy ∑ i ∈ Π j sign( b i ) · a i > ε · | Π j | . The validity of such a certificate can be verified in polynomial time by checking each of these conditions directly.

It remains to show that such a certificate exists if and only if the instance is affirmative. If prank ε ( w ) ≤ r , then there exists a parameter w /star ∈ ¯ B ∞ ( w ; ε ) with rank( w /star ) ≤ r . The partition computed from Stage 2 of COMPRESS ( w /star ) satisfies the required properties for w ∈ ¯ B ∞ ( w /star ; ε ) .

Conversely, given such a partition, for each Π j , define v j ∈ R 2 as the centroid of the bounding rectangle of the set of points { sign( b i ) · ( b i , c i ) : i ∈ Π j } , that is,

All of the points within these bounding rectangles are at most uniform distance ε from their centroids. To construct a nearby low-rank parameter, follow the proof of Theorem 5.1 using Π 1 , . . . , Π J and v 1 , . . . , v J in place of their namesakes from Algorithms 5.1 and 5.2. Thus prank ε ( w ) ≤ r .

7 Discussion

In this paper, we have studied losslessly compressible neural network parameters, measuring the size of a network by the number of hidden units. Losslessly compressible parameters comprise a measure zero subset of the parameter space, but this is a rich subset that stretches throughout the entire parameter space (Anonymous, 2023). Moreover, the neighbourhood of this region has nonzero measure and comprises approximately compressible parameters.

It's possible that part of the empirical success of deep learning can be explained by the proximity of learned neural networks to losslessly compressible parameters. Our theoretical and algorithmic contributions, namely the notions of rank and proximate rank and their associated algorithms, serve as a foundation for future research in this direction. In this section, we outline promising next steps for future work and discuss limitations of our approach.

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

Limitations of the lossless compressibility framework. Section 4 offers efficient algorithms for optimal lossless compression and computing the rank of neural network parameters. However, the rank is an idealised notion, serving as a basis for the theory of proximate rank. One would not expect to find compressible parameters in practice, since numerical imprecision is likely to prevent the observation of identically equal, negative, or zero weights in practice. Moreover, the number of units is not the only measure of a network's description length. For example, the sparsity and precision of weights may be relevant axes of parsimony in neural network modelling.

Returning to the deep learning context-there is a gap between lossless compressibility and phenomena of approximate compressibility. In practical applications and empirical investigations, the neural networks in question are only approximately preserved the function, and moreover the degree of approximation may deteriorate for unlikely inputs. Considering the neighbourhoods of losslessly compressible parameters helps bridge this gap, but there are approximately compressible neural networks beyond the proximity of losslessly compressible parameters, which are not accounted for in this approach. More broadly, a comprehensive account of neural network compressibility must consider architectural redundancy as well as redundancy in the parameter.

Tractable detection of proximity to low-rank parameters. An important direction for future work is to empirically investigate the proximity of low-rank neural networks to the neural networks that arise during the course of successful deep learning. Unfortunately, our main result (Theorem 6.2) suggests that detecting such proximity is computationally intractable in general, due to the complex structure of the neighbourhoods of low-rank parameters.

There is still hope for empirically investigating the proximate rank of learned networks. Firstly, NP -completeness does not preclude efficient approximation algorithms, and approximations are still useful as a one-sided test of proximity to low-rank parameters. Algorithm 5.1 provides a naive approximation, with room for improvement in future work. Secondly, Theorem 6.2 is a worst-case analysis-Section 6 essentially constructs pathological parameters poised between nearby low-rank regions such that choosing the optimal direction of perturbation involves solving (a hard instance of) Boolean satisfiability. Such instances might be rare in practice (cf. the related problem of k -means clustering; Daniely et al., 2012). As an extreme example, detecting proximity to merely compressible parameters ( r = h -1 ) permits a polytime solution based on the reducibility conditions.

Towards lossless compressibility theory in modern architectures. We have studied lossless compressibility in the simple, concrete setting of single-hidden-layer hyperbolic tangent networks. Several elements of our approach will be useful for future work on more modern architectures. At the core of our analysis are structural redundancies arising from zero, constant, or proportional units (cf. reducibility conditions (i)-(iii)). In particular, the computational difficulty of bounding the proximate rank is due to the approximate merging embedding a hard clustering problem. These features are not due to the specifics of the hyperbolic tangent, rather they are generic features of any layer in a feed-forward network component.

In more complex architectures there will be additional or similar opportunities for compression. While unit negation symmetries are characteristic of odd nonlinearities, other nonlinearities will exhibit their own affine symmetries which can be handled analogously. Further redundancies will arise from interactions between layers or from specialised computational structures.

8 Conclusion

Towards a better understanding of complexity and compressibility in learned neural networks, we have developed a theoretical and algorithmic framework for lossless compressibility in singlehidden-layer hyperbolic tangent networks. The rank is a measure of a parameter's lossless compressibility. Section 4 offers efficient algorithms for performing optimal lossless compression and computing the rank. The proximate rank is a measure of proximity to low-rank parameters. Section 5 offers an efficient algorithm for approximately bounding the proximate rank. In Section 6, we show that optimally bounding the proximate rank, or, equivalently, detecting proximity to low-rank parameters, is NP -complete, by reduction from Boolean satisfiability via a novel hard clustering problem. These results underscore the complexity of losslessly compressible regions of the parameter space and lay a foundation for future theoretical and empirical work on detecting losslessly compressibile parameters arising while learning with more complex architectures.

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

References

Francesca Albertini, Eduardo D. Sontag, and Vincent Maillot. Uniqueness of weights for neural networks. In Artificial Neural Networks for Speech and Vision , pages 113-125. Chapman & Hall, London, 1993 . Proceedings of a workshop held at Rutgers University in 1992. Access via Eduardo D. Sontag. Cited on page 2.

Shun-ichi Amari, Hyeyoung Park, and Tomoko Ozeki. Singularities affect dynamics of learning in neuromanifolds. Neural Computation , 18(5):1007-1065, 2006 . Access via Crossref. Cited on page 2.

Shun-ichi Amari, Tomoko Ozeki, Ryo Karakida, Yuki Yoshida, and Masato Okada. Dynamics of learning in MLP: Natural gradient and singularity revisited. Neural Computation , 30(1):1-33, 2018 . Access via Crossref. Cited on page 2.

Anonymous. Functional equivalence and path connectivity of reducible hyperbolic tangent networks. 2023 . Anonymised article included with supplementary material. Cited on pages 1, 2, 3, 8, and 26.

Caglar Aytekin, Francesco Cricri, and Emre Aksu. Compressibility loss for neural network weights. 2019 . Preprint arXiv:1905.01044 [cs.LG]. Cited on page 3.

Piotr Berman, Alex D. Scott, and Marek Karpinski. Approximation hardness and satisfiability of bounded occurrence instances of SAT. Technical Report IHES/M/03/25, Institut des Hautes Études Scientifiques [Institute of Advanced Scientific Studies], 2003 . Access via CERN. Cited on page 18.

Cristian Buciluˇ a, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 535-541. ACM, 2006 . Access via Crossref. Cited on page 1.

Stephen Casper, Xavier Boix, Vanessa D'Amario, Ling Guo, Martin Schrimpf, Kasper Vinken, and Gabriel Kreiman. Frivolous units: Wider networks are not really that wide. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence , volume 8, pages 6921-6929. AAAI Press, 2021 . Access via Crossref. Cited on page 3.

Márcia R. Cerioli, Luerbio Faria, Talita O. Ferreira, and Fábio Protti. On minimum clique partition and maximum independent set on unit disk graphs and penny graphs: Complexity and approximation. Electronic Notes in Discrete Mathematics , 18:73-79, 2004 . Access via Crossref. Cited on pages 6, 15, 18, and 26.

Márcia R. Cerioli, Luerbio Faria, Talita O. Ferreira, and Fábio Protti. A note on maximum independent sets and minimum clique partitions in unit disk graphs and penny graphs: Complexity and approximation. RAIRO: Theoretical Informatics and Applications , 45(3):331-346, 2011 . Access via Crossref. Cited on pages 6, 15, 18, and 26.

An Mei Chen, Haw-minn Lu, and Robert Hecht-Nielsen. On the geometry of feedforward neural network error surfaces. Neural Computation , 5(6):910-927, 1993 . Access via Crossref. Cited on page 1.

Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine , 35(1):126-136, 2018 . Access via Crossref. Cited on pages 3 and 10.

Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. 2020 . Preprint arXiv:1710.09282v9 [cs.LG]. Updated version of Cheng et al. (2018). Cited on page 3.

Tejalal Choudhary, Vipul Mishra, Anurag Goswami, and Jagannathan Sarangapani. A comprehensive survey on model compression and acceleration. Artificial Intelligence Review , 53(7):51135155, 2020 . Access via Crossref. Cited on page 3.

Brent N. Clark, Charles J. Colbourn, and David S. Johnson. Unit disk graphs. Discrete Mathematics , 86(1-3):165-177, 1990 . Access via Crossref. Cited on page 15.

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

Stephen A. Cook. The complexity of theorem-proving procedures. In Proceedings of the Third Annual ACM Symposium on Theory of Computing , pages 151-158. ACM, 1971 . Access via Crossref. Cited on pages 3, 6, 17, and 18.

Florent Cousseau, Tomoko Ozeki, and Shun-ichi Amari. Dynamics of learning in multilayer perceptrons near singularities. IEEE Transactions on Neural Networks , 19(8):1313-1328, 2008 . Access via Crossref. Cited on page 2.

Amit Daniely, Nati Linial, and Michael Saks. Clustering is difficult only when it does not matter. 2012 . Preprint arXiv:1205.4891 [cs.LG]. Cited on page 9.

Charles Fefferman. Reconstructing a neural net from its output. Revista Matemática Iberoamericana , 10(3):507-555, 1994 . Access via Crossref. Cited on pages 1 and 2.

Charles Fefferman and Scott Markel. Recovering a feed-forward net from its output. In Advances in Neural Information Processing Systems 6 , pages 335-342. Morgan Kaufmann, 1993 . Access via NeurIPS. Cited on page 2.

Kenji Fukumizu. A regularity condition of the information matrix of a multilayer perceptron network. Neural Networks , 9(5):871-879, 1996 . Access via Crossref. Cited on pages 1 and 26.

Kenji Fukumizu and Shun-ichi Amari. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural Networks , 13(3):317-327, 2000 . Access via Crossref. Cited on page 2.

Kenji Fukumizu, Shoichiro Yamaguchi, Yoh-ichi Mototake, and Mirai Tanaka. Semi-flat minima and saddle points by embedding neural networks to overparameterization. In Advances in Neural Information Processing Systems 32 , pages 13868-13876. Curran Associates, 2019 . Access via NeurIPS. Cited on page 2.

Jesus Garcia-Diaz, Jairo Sanchez-Hernandez, Ricardo Menchaca-Mendez, and Rolando MenchacaMendez. When a worse approximation factor gives better performance: A 3-approximation algorithm for the vertex k -center problem. Journal of Heuristics , 23(5):349-366, 2017 . Access via Crossref. Cited on page 15.

Michael R. Garey and David S. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness . W. H. Freeman and Company, 1979 . Cited on pages 3 and 26.

L. Hakimi. Optimum locations of switching centers and the absolute centers and medians of a graph. Operations Research , 12(3):450-459, 1964 . Access via Crossref. Cited on pages 6 and 15.

Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory , pages 5-13. ACM, 1993 . Access via Crossref. Cited on page 3.

Geoffrey E. Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. Presented at Twenty-eighth Conference on Neural Information Processing Systems, Deep Learning workshop, 2014 . Preprint arXiv:1503.02531 [stat.ML]. Cited on page 1.

Klaus Jansen and Haiko Müller. The minimum broadcast time problem for several processor networks. Theoretical Computer Science , 147(1-2):69-85, 1995 . Access via Crossref. Cited on page 18.

Kariv and S. L. Hakimi. An algorithmic approach to network location problems. I: the p -centers. SIAM Journal on Applied Mathematics , 37(3):513-538, 1979 . Access via Crossref. Cited on pages 6 and 15.

Richard M. Karp. Reducibility among combinatorial problems. In Complexity of Computer Computations , pages 85-103. Springer, 1972 . Access via Crossref. Cited on pages 15 and 26.

Vˇ era K˚ urková and Paul C. Kainen. Functionally equivalent feedforward neural networks. Neural Computation , 6(3):543-558, 1994 . Access via Crossref. Cited on page 2.

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

Leonid A. Levin. Universal sequential search problems. Problemy Peredachi Informatsii [Problems of Information Transmission] , 9(3):115-116, 1973 . In Russian. Translated into English in ? . Cited on pages 3, 6, and 17.

David Lichtenstein. Planar formulae and their uses. SIAM Journal on Computing , 11(2):329-343, 1982 . Access via Crossref. Cited on page 18.

Yanpei Liu, Aurora Morgana, and Bruno Simeone. A linear algorithm for 2-bend embeddings of planar graphs in the two-dimensional grid. Discrete Applied Mathematics , 81(1-3):69-91, 1998 . Access via Crossref. Cited on pages 19 and 24.

Meena Mahajan, Prajakta Nimbhorkar, and Kasturi Varadarajan. The planar k -means problem is NPhard. Theoretical Computer Science , 442:13-21, 2012 . Access via Crossref. Cited on pages 6 and 15.

Mary Phuong and Christoph H. Lampert. Functional vs. parametric equivalence of ReLU networks. In 8th International Conference on Learning Representations . OpenReview, 2020 . Access via OpenReview. Cited on pages 1 and 2.

Piziak and P. L. Odell. Full rank factorization of matrices. Mathematics Magazine , 72(3):193201, 1999 . Access via Crossref. Cited on page 4.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. Presented at the Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing, 2019 . Preprint arXiv:1910.01108 [cs.CL]. Cited on page 1.

Berfin ¸ Sim¸ sek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In Proceedings of the 38th International Conference on Machine Learning , pages 9722-9732. PMLR, 2021 . Access via PMLR. Cited on page 2.

Kenneth J. Supowit. Topics in Computational Geometry . Ph.D. thesis, University of Illinois at Urbana-Champaign, 1981 . Access via ProQuest. Cited on pages 6 and 15.

Héctor J. Sussmann. Uniqueness of the weights for minimal feedforward nets with a given inputoutput map. Neural Networks , 5(4):589-593, 1992 . Access via Crossref. Cited on pages 1, 2, 3, 4, and 26.

Taiji Suzuki, Hiroshi Abe, Tomoya Murata, Shingo Horiuchi, Kotaro Ito, Tokuma Wachi, So Hirai, Masatoshi Yukishima, and Tomoaki Nishimura. Spectral pruning: Compressing deep neural networks via spectral analysis and its generalization error. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence , pages 2839-2846. IJCAI, 2020 a. Access via Crossref. Cited on page 3.

Taiji Suzuki, Hiroshi Abe, and Tomoaki Nishimura. Compression based bound for non-compressed network: Unified generalization error analysis of large compressible deep neural network. In 8th International Conference on Learning Representations . OpenReview, 2020 b. Access via OpenReview. Cited on page 3.

Craig A. Tovey. A simplified NP-complete satisfiability problem. Discrete Applied Mathematics , 8(1):85-89, 1984 . Access via Crossref. Cited on page 18.

Leslie G. Valiant. Universality considerations in VLSI circuits. IEEE Transactions on Computers , 100(2):135-140, 1981 . Access via Crossref. Cited on pages 6 and 19.

Verner Vlaˇ ci´ c and Helmut Bölcskei. Affine symmetries and neural network identifiability. Advances in Mathematics , 376:107485, 2021 . Access via Crossref. Cited on page 2.

Verner Vlaˇ ci´ c and Helmut Bölcskei. Neural network identifiability for a family of sigmoidal nonlinearities. Constructive Approximation , 55(1):173-224, 2022 . Access via Crossref. Cited on page 2.

524

525

526

527

528

529

530

531

532

533

534

Sumio Watanabe. Algebraic Geometry and Statistical Learning Theory . Cambridge University Press, 2009 . Cited on page 1.

Haikun Wei, Jun Zhang, Florent Cousseau, Tomoko Ozeki, and Shun-ichi Amari. Dynamics of learning near singularities in layered networks. Neural Computation , 20(3):813-843, 2008 . Access via Crossref. Cited on page 2.

Susan Wei, Daniel Murfet, Mingming Gong, Hui Li, Jesse Gell-Redman, and Thomas Quella. Deep learning is singular, and that's good. IEEE Transactions on Neural Networks and Learning Systems , 2022 . Access via Crossref. To appear in an upcoming volume. Cited on page 1.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations . OpenReview, 2017 . Access via OpenReview. Cited on pages 1 and 13.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding 535 deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107536 115, 2021 . Access via Crossref. Republication of Zhang et al. (2017). Cited on page 1. 537

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the replication of the same function using a smaller network. It provides a procedure for achieving optimal lossless compression in the context of single-hidden-layer hyperbolic tangent networks.\n\nThis paper introduces the idea of 'rank' of a parameter, defined as the least number of hidden units needed to replicate the same function. The paper further defines the 'proximate rank' of a parameter as the rank of the most compressible parameter within a limited L-inf neighborhood.\n\nThe paper also demonstrates that estimating the proximate rank is an NP-complete problem through a reduction from Boolean satisfiability using a geometric problem that involves covering points in a plane with small squares.\n\nSTRENGTHS:\n1. This is a theoretical treatment of a very important problem and the results are interesting.\n\n2. The paper shows that the problem of bounding the approximate rank below a given value is an NP-complete decision problem.\n\n3. Paper is well presented and clear.\n\nWEAKNESSES:\n1. The results do not seem to be important. Specifically, does anyone care if instead of totally lossless compression, we have an infinitesimal error in representation?\n\n2. The algorithm given in this paper is only applicable to single-hidden layer hyperbolic tangent networks, without obvious ways of extending it to more general cases.\n\n3. The paper aims to provide a theoretical ground for network compression, but did not argue extensively on its connection on the empirical success of existing network compression literature.\n\nQUESTIONS:\n1. Is it possible to define a  more relaxed version of \"rank\", that is more applicable to the existing neural network compression methods? \n\n2. There has been some theoretical treatments of the problem of network compression. For instance, https://arxiv.org/abs/2206.05604 approach the problem with linear approximation. What are the connections to your work?\n\nLIMITATIONS:\n1. In practice it is allowed that the compressed networks have slight performance drop compared to the original network, but this paper does not consider this problem.\n\n2. The proposed algorithm only considers the single-hidden layer hyperbolic tangent networks, which is not often employed in practice.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the idealized phenomenon of lossless compressibility, whereby an identical\nfunction can be implemented with a smaller network, in the setting of single-hidden-layer hyperbolic tangent networks.  It introduces the notion of rank as the minimal number of hidden units (used to measure the \"size\" of a network) required to implement a given NN function and a constructive algorithm for computing it, as well as the notion of *proximate rank* as the rank of the most compressible parameter vector within a small L_infty norm ball and an algorithm for upper-bounding it. The paper further shows that bounding the proximate rank of a given parameter vector is an NP-complete problem.\n\nSTRENGTHS:\nWhile I'm not an expert on algorithms and cannot comment on correctness or the novelty of the approach, \nI followed the high-level argument and believe the contribution is significant. The paper studies a fundamental aspect of deep learning, i.e., the compressibility and description length of neural networks, which is mostly dominated by empirical research and demands better theoretical understanding. The paper goes beyond the classical setting of lossless compressibility (c.f. Sussmann 1992) by introducing the notion of proximate rank and proving a basic hardness results for it, which will hopefully lay a foundation for future studies of this topic.\n\nWEAKNESSES:\nSection 6 (Computational complexity of proximate rank) is a bit hard to follow for non-experts. One possibility could be to include a discussion at the beginning of Section 6 that summarizes the proof at a high-level.\n\nQUESTIONS:\n1. I'm curious how the authors define \"description length\" (\"polynomial description length\" in footnote 8). It appears there's a widely accepted notion of description length in this line of work? \n\n2. Related to the above, could the authors comment on how their notion of NN size (minimum number of hidden units) relates to alternative description length such as Kolgomorov complexity?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose two notions for studying neural network complexity with, on single hidden layer neural networks. The first is _rank_, which corresponds to the smallest number of parameters that can produce a network that is functionally equivalent to the original, and the second is _proximate rank_, which is the rank of the network with smallest rank in an $L^\\inf$ neighborhood of the original model('s parameters). The authors propose an algorithm for computing rank, and propose a heuristic algorithm for upper bounding proximate rank. They also show that exactly computing proximate rank is NP-complete, so any future study will have to rely on approximations / bounds on this quantity.\n\nSTRENGTHS:\n- The notions the authors introduce are interesting and future work can indeed utilize them in interesting ways. The metrics they propose importantly diverge from metrics such as $\\ell_p$-compressibility in that they are defined based on functional equivalence, which might imply them being more useful for future learning theoretic research.\n- The paper is very well structured, and the exposition is clear and easy to follow. Connections to previous literature are made clear (but see below).\n\nWEAKNESSES:\nThe paper is beset by two important problems:\n- As the authors widely acknowledge, their notions are defined on single layer neural networks with tanh activation, with no clear map for generalization to more realistic setups. It would be valuable to see a roadmap for this, as it would help evaluate whether proximate rank is a potentially useful theoretical notion for future research or in the worst case a \"dead end\" in terms of applicability to realistic ML models.\n- The authors do not discuss how or why their notions of rank and proximate rank could be useful for future research on generalization in deep learning (or robustness etc.). Given the venue the paper is submitted, I am assuming that the authors believe that these results might have learning theoretical implications down the line, but they are mostly silent on this issue. They correctly observe that approximate compressibility research has been utilized for such aims, but do not opine on how their alternative notions should be superior or even have qualitatively different contributions to such research. While this is not a problem per se in general, it is more so for a machine learning conference.\n\nQUESTIONS:\n- L68: Is there a common theoretical / practical reason for previous literature having given little consideration to cases other than \"minimal networks\"?\n- L75: What are canonical equivalent parameters?\n- I think a broader discussion of the relation of current work to \"compressibility\" research would be helpful for the reader, especially so for the generalization-related subset of this research, e.g. [1] being one of the most well-known.\n\n[1] Arora, S., Ge, R., Neyshabur, B., & Zhang, Y. (2018). Stronger Generalization Bounds for Deep Nets via a Compression Approach. ICLR.\n\nLIMITATIONS:\nThe authors are transparent about the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deep-dives into the concept of lossless network compressibility. It presents an algorithm for optimal lossless compression in single-hidden-layer hyperbolic tangent networks, which can in part generalize to other more relevant feedforward architectures. The authors introduce the novel concept of \"proximate rank\", which measures network complexity, and demonstrate that bounding the proximate rank is an NP-complete problem, thereby suggesting that the problem of finding highly compressible networks is very hard.\n\nSTRENGTHS:\n- originality: The paper introduces novel approaches to obtain lossless compression and introduces the novel concept of proximate rank as a measure of neural network complexity. Furthermore the paper also introduces novel techniques to prove that bounding the proximate rank measure is an NP-complete problem. In summary, many novel contributions which, as the authors argue, lay the foundations for future work identifying losslessly compressible parameters in deep learning structures.   \n\n- quality: The quality of the paper seems sound. The topic is a bit away from my area of expertise and I did not check the math in detail though.\n\n- clarity: the paper is very well written. \n\n- significance: Lossless compressibility is a fundamental topic in deep learning, particularly since most recent state-of-the-art research involve very large foundation models. Having a theoretical framework and some robust approaches to measure network complexity and compressibility are key to advance the field of deep learning and the applicability of modern architectures.\n\nWEAKNESSES:\nSingle-hidden-layer hyperbolic tangent network can be a good subject to start developing a theoretical framework and a set of tools to measure and optimize network complexity and compressibility, but as the authors point out, this architecture is of little relevance otherwise (for current research). This issue, which is raised by the authors, is a minor weakness but it does limit the potential impact of the present work.\n\nQUESTIONS:\nNo questions for the authors.\n\nLIMITATIONS:\nAuthors have addressed all the limitations clearly.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}