Neural Network Approximation based on Hausdorff distance of Zonotopes

In this work we theoretically contribute to neural network approximation by providing a novel tropical geometrical viewpoint to structured neural network compression . In particular , we show that the approximation error between two neural networks with ReLU activations and one hidden layer depends on the Hausdorff distance of the tropical zonotopes of the networks . This theorem comes as a first step towards a purely geometrical interpretation of neural network approximation . Based on this theoretical contribution , we propose geometrical methods that employ the K-means algorithm to compress the fully connected parts of ReLU activated deep neural networks . We analyze the error bounds of our algorithms theoretically based on our approximation theorem and evaluate them empirically on neural network compression . Our experiments follow a proof-of-concept strategy and indicate that our geometrical tools achieve improved performance over relevant tropical geometry techniques and can be competitive against non-tropical methods . 1 INTRODUCTION . Tropical geometry ( Maclagan & Sturmfels , 2015 ) is a mathematical field based on algebraic geometry and strongly linked to polyhedral and combinatorial geometry . It is built upon the tropical semiring which originally refers to the min-plus semiring ( Rmin , ∧ , + ) , but may also refer to the max-plus semiring ( Cuninghame-Green , 2012 ; Butkovič , 2010 ) . In our work , we follow the convention of the max-plus semiring ( Rmax , ∨ , + ) which replaces the classical operations of addition and multiplication by max and sum respectively . These operations turn polynomials into piecewise linear functions and , thus , having direct applications in neural networks . Tropical mathematics have a wide range of applications including game theory ( Akian et al. , 2012 ) , dynamical systems on weighted lattices ( Maragos , 2017 ) , finite state transducers ( Theodosis & Maragos , 2018 ; 2019 ) and convex regression ( Maragos & Theodosis , 2020 ; Tsilivis et al. , 2021 ) . Recently , there has been remarkable theoretical impact of tropical geometry in the study of neural networks and machine learning ( Maragos et al. , 2021 ) . Zhang et al . ( 2018 ) prove the equivalence of ReLU activated neural networks with tropical rational mappings . Furthermore , they show that zonotopes may serve as a computational mean for applications such as computing the bound shown in ( Montúfar et al. , 2014 ) regarding the number of the network ’ s linear regions . In a similar context , Charisopoulos & Maragos ( 2018 ) compute an upper bound to the number of linear regions of convolutional and MaxOut layers and propose a randomized algorithm for linear region counting . Moreover , Charisopoulos & Maragos ( 2017 ) study the morphological perceptron tropically and propose a training algorithm using the convex-concave procedure . Also , Dimitriadis & Maragos ( 2021 ) examine the training , pruning and shape properties of Morphological neural networks . Pruning ( Blalock et al. , 2020 ) or , generally , compressing neural networks gained interest in recent years due to the surprising capability of reducing the size of a neural network without compromising performance . As tropical geometry explains the mathematical structure of neural networks , pruning may be revisited tropically . Indeed , Alfarra et al . ( 2020 ) study the lottery ticket hypothesis ( Frankle & Carbin , 2018 ) from the tropical geometrical perspective and propose an unstructured compression algorithm based sparsifying the zonotope matrices of the network . Further , Smyrnis et al . ( 2020 ) propose a novel tropical division algorithm that applies to neural network minimization . A generalization of this method applies to multiclass networks ( Smyrnis & Maragos , 2020 ) . Contributions In our work , we contribute to structured neural network approximation from the mathematical viewpoint of tropical geometry : • We establish a novel bound on the approximation error between two neural networks with ReLU activations and one hidden layer . To prove this we bound the difference of the networks ’ tropical polynomials via the Hausdorff distance of their respective zonotopes . • Motivated by the geometrical bound , we derive two geometrical neural network compression methods that are based on zonotope reduction and employ K-means algorithm for clustering . Our algorithms are applied on the fully connected layers of ReLU activated neural networks . • Our geometrical algorithms are analyzed both theoretically and experimentally . The theoretical evaluation is based on the bounding theorem of neural network approximation error by the Hausdorff distance of the zonotopes . On the experimental part , we examine the performance of our algorithms on retaining the accuracy of convolutional neural networks when applying compression on their fully connected layers . Proofs , illustrations and experimental specifications are included in the supplementary material . 2 BACKGROUND ON TROPICAL GEOMETRY . We study tropical geometry from the viewpoint of the max-plus semiring ( Rmax , ∨ , + ) which is defined as the set Rmax = R ∪ { −∞ } equipped with two operations ( ∨ , + ) . Operation ∨ stands for max and + stands for sum . In max-plus algebra we define polynomials in the following way . Tropical polynomials A tropical polynomial f in d variables x = ( x1 , x2 , ... , xd ) T is defined as the function f ( x ) = max a∈A { aTx+ ca } ( 1 ) where A is a finite collection of vectors in Rd and ca is the corresponding monomial coefficient in Rmax = R ∪ { −∞ } . The set of such polynomials constitutes the semiring Rmax [ x ] of tropical polynomials . Note that each term aTx + ca corresponds to a hyperplane in Rd . We thus call the vectors a ∈ A the slopes of the tropical polynomial , and ca the respective bias . We allow slopes to be vectors with real coefficients rather than integer ones , as it is normally the case for regular polynomials . These polynomials are also referred to as signomials ( Duffin & Peterson , 1973 ) in the literature . Polytopes Polytopes have been studied extensively ( Ziegler , 2012 ; Grünbaum , 2013 ) and occur as a geometric tool for fields such as linear programming and optimization . They also have an important role in the analysis of neural networks . For instance , Zhang et al . ( 2018 ) ; Charisopoulos & Maragos ( 2018 ) show that linear regions of neural networks correspond to vertices of polytopes . Thus , the counting of linear regions reduces to a combinatorial geometry problem . In what follows , we explore this connection of tropical geometry with polytopes . Consider the tropical polynomial defined in ( 1 ) . The Newton Polytope associated to f ( x ) is defined as the convex hull of the slopes of the polynomial Newt ( f ) : = conv { a : a ∈ A } Furthermore , the Extended Newton Polytope of f ( x ) is defined as the convex hull of the slopes of the polynomial extended in the last dimension by the corresponding bias coefficient . ENewt ( f ) : = conv { ( aT , ca ) : a ∈ A } The following proposition computes the Extended Newton Polytope that occurs when a tropical operation is applied between two tropical polynomials . It will allow us to compute the polytope representation corresponding to a neural network ’ s hidden layer . Proposition 1 . ( Zhang et al. , 2018 ; Charisopoulos & Maragos , 2018 ) Let f , g ∈ Rmax [ x ] be two tropical polynomials . Then for the Extended Newton Polytopes it is true that ENewt ( f ∨ g ) = conv { ENewt ( f ) ∪ ENewt ( g ) } ENewt ( f + g ) = ENewt ( f ) ⊕ ENewt ( g ) Here ⊕ denotes Minkowski addition . In particular , for two sets A , B ⊆ Rd it is defined as A⊕B : = { a+ b |a ∈ A , b ∈ B } Corollary 1 . This result can be generalized to any finite set of polynomials using induction . Example 1 . Let f , g be two tropical polynomials in 2 variables , such that f ( x , y ) = max ( 2x+ y + 1 , 0 ) , g ( x , y ) = max ( x , y , 1 ) The tropical operations applied to these polynomials give f ∨ g = max ( 2x+ y + 1 , 0 , x , y , 1 ) f + g = max ( 3x+ y + 1 , x , 2x+ 2y + 1 , y , 2x+ y + 2 , 1 ) Fig . 1 illustrates the Extended Newton Polytopes of the original and the computed polynomials . The Extended Newton Polytope provides a geometrical representation of a tropical polynomial . In addition , it may be used to compute the values that the polynomial attains , as Proposition 2 indicates . Proposition 2 . ( Charisopoulos & Maragos , 2018 ) Let f ∈ Rmax [ x ] be a tropical polynomial in d variables . Let UF ( ENewt ( f ) ) be the points in the upper envelope of ENewt ( f ) , where upward direction is taken regarding the last dimension of Rd+1 . Then for each a ∈ A there exists a linear region of f on which f ( x ) = aTx+ ca if and only if ( aT , ca ) is a vertex of UF ( ENewt ( f ) ) . Example 2 . Using the polynomials from Example 1 we compute a reduced representation for f ∨ g. f ∨ g = max ( 2x+ y + 1 , 0 , x , y , 1 ) = max ( 2x+ y + 1 , x , y , 1 ) Indeed , the remaining terms correspond to the vertices of UF ( ENewt ( f ∨ g ) ) shown in Fig . 1 . 2.1 TROPICAL GEOMETRY OF NEURAL NETWORKS . Tropical geometry has the capability of expressing the mathematical structure of ReLU activated neural networks . We review some of the basic properties governing neural networks and introduce notation that will be used in our analysis . For this purpose , consider the ReLU activated neural network of Fig . 2 with one hidden layer . Network tropical equations The network consists of an input layer x = ( x1 , ... , xd ) , a hidden layer f = ( f1 , ... , fn ) with ReLU activations , an output layer v = ( v1 , ... , vm ) and two linear layers defined by the matrices A , C respectively . As illustrated in Fig . 2 we have Ai , : = ( aTi , bi ) for the first linear layer and Cj , : = ( cj1 , cj2 , ... , cjn ) for the second linear layer , as we ignore its biases . Furthermore , the output of the i−th component of the hidden layer f is computed as fi ( x ) = max ( d∑ k=1 aikxk + bi , 0 ) = max ( aTi x+ bi , 0 ) ( 2 ) We deduce that each fi is a tropical polynomial with two terms . It therefore follows that ENewt ( fi ) is a linear segment in Rd+1 . The components of the output layer may be computed as vj ( x ) = n∑ i=1 cjifi ( x ) = ∑ cji > 0 |cji|fi ( x ) − ∑ cji < 0 |cji|fi ( x ) = pj ( x ) − qj ( x ) ( 3 ) Tropical rational functions Functions pj , qj are both linear combinations of { fi } with positive coefficients , which implies that they are tropical polynomials . We conclude that every output node vi can be written as a difference of two tropical polynomials , which is defined as a tropical rational function . This indicates that the output layer of the neural network of Fig . 2 is equivalent to a tropical rational mapping . In fact , this result holds for deeper networks , in general , as demonstrated by Theorem 1 . Theorem 1 . ( Zhang et al. , 2018 ) Consider a ReLU activated deep neural network as a mapping F : Rd → Rm . Then F is equivalent to a tropical rational mapping . It is not known whether a tropical rational function r ( x ) admits an efficient geometric representation that determines its values { r ( x ) } for x ∈ Rd , as it holds for tropical polynomials with their polytopes in Proposition 2 . For this reason , we choose to work separately on the polytopes of the tropical polynomials pj , qj . Zonotopes Zonotopes are defined as the Minkowski sum of a finite set of line segments . They are a special case of polytopes that occur as a building block for our network . These geometrical structures provide a representation of the polynomials pj , qj in ( 3 ) that further allows us to build our compression algorithms . We use the notation Pj , Qj for the Extended Newton Polytopes of tropical polynomials pj , qj , respectively . Notice from ( 3 ) that for each component vj of the output pj , qj are written as linear combinations of tropical polynomials that correspond to linear segments . Thus Pj and Qj are zonotopes . We call Pj the positive zonotope , corresponding to the positive polynomial pj and Qj the negative one . Zonotope Generators Each neuron of the hidden layer represents geometrically a line segment contributing to the positive or negative zonotope . We thus call these line segments generators of the zonotope . The generators further receive the characterization positive or negative depending on the zonotope they contribute to . It is intuitive to expect that a zonotope gets more complex as its number of generators increases . In fact , each vertex of the zonotope can be computed as the sum of vertices of the generators , where we choose a vertex from each generating line segment , either 0 or cji ( aTi , bi ) . We summarize the above with the following extension of ( Charisopoulos & Maragos , 2018 ) . Proposition 3 . Pj , Qj are zonotopes in Rd+1 . For each vertex v of Pj there exists a subset of indices I+ of { 1 , 2 , ... , n } with cji > 0 , ∀i ∈ I+ such that v = ∑ i∈I+ cji ( aTi , bi ) . Similarly , a vertex u of Qj can be written as u = ∑ i∈I− cji ( aTi , bi ) where I− corresponds to cji < 0 , ∀i ∈ I− .

Summary: This paper proposes to study neural network compression through the lens of zonotope reduction. They present approximation bounds for tropical polynomials based on Hausdorff distance and use this to motivate the development of two neural network compression methods. These methods are evaluated on the MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets.