From Hard to Soft: Understanding Deep Network Nonlinearities via Vector Quantization and Statistical Inference

1 INTRODUCTION . Deep ( neural ) networks ( DNs ) have recently come to the fore in a wide range of machine learning tasks , from regression to classification and beyond . A DN is typically constructed by composing a large number of linear/affine transformations interspersed with up/down-sampling operations and simple scalar nonlinearities such as the ReLU , absolute value , sigmoid , hyperbolic tangent , etc . Goodfellow et al . ( 2016 ) . Scalar nonlinearities are crucial to a DN ’ s performance . Indeed , without nonlinearity , the entire network would collapse to a simple affine transformation . But to date there has been little progress understanding and unifying the menagerie of nonlinearities , with few reasons to choose one over another other than intuition or experimentation . Recently , progress has been made on understanding the rôle played by piecewise affine and convex nonlinearities like the ReLU , leaky ReLU , and absolute value activations and downsampling operations like max- , average- , and channel-pooling Balestriero & Baraniuk ( 2018a ; b ) . In particular , these operations can be interpreted as max-affine spline operators ( MASOs ) Magnani & Boyd ( 2009 ) ; Hannah & Dunson ( 2013 ) that enable a DN to find a locally optimized piecewise affine approximation to the prediction operator given training data . A spline-based prediction is made in two steps . First , given an input signal x , we determine which region of the spline ’ s partition of the domain ( the input signal space ) it falls into . Second , we apply to x the fixed ( in this case affine ) function that is assigned to that partition region to obtain the prediction ŷ = f ( x ) . The key result of Balestriero & Baraniuk ( 2018a ; b ) is any DN layer constructed from a combination of linear and piecewise affine and convex is a MASO , and hence the entire DN is merely a composition of MASOs . MASOs have the attractive property that their partition of the signal space ( the collection of multidimensional “ knots ” ) is completely determined by their affine parameters ( slopes and offsets ) . This provides an elegant link to vector quantization ( VQ ) and K-means clustering . That is , during learning , a DN implicitly constructs a hierarchical VQ of the training data that is then used for splinebased prediction . This is good progress for DNs based on ReLU , absolute value , and max-pooling , but what about DNs based on classical , high-performing nonlinearities that are neither piecewise affine nor convex like the sigmoid , hyperbolic tangent , and softmax or fresh nonlinearities like the swish Ramachandran et al . ( 2017 ) that has been shown to outperform others on a range of tasks ? Contributions . In this paper , we address this gap in the DN theory by developing a new framework that unifies a wide range of DN nonlinearities and inspires and supports the development of new ones . The key idea is to leverage the yinyang relationship between deterministic VQ/K-means and probabilistic Gaussian Mixture Models ( GMMs ) Biernacki et al . ( 2000 ) . Under a GMM , piecewise affine , convex nonlinearities like ReLU and absolute value can be interpreted as solutions to certain natural hard inference problems , while sigmoid and hyperbolic tangent can be interpreted as solutions to corresponding soft inference problems . We summarize our primary contributions as follows : Contribution 1 : We leverage the well-understood relationship between VQ , K-means , and GMMs to propose the Soft MASO ( SMASO ) model , a probabilistic GMM that extends the concept of a deterministic MASO DN layer . Under the SMASO model , hard maximum a posteriori ( MAP ) inference of the VQ parameters corresponds to conventional deterministic MASO DN operations that involve piecewise affine and convex functions , such as fully connected and convolution matrix multiplication ; ReLU , leaky-ReLU , and absolute value activation ; and max- , average- , and channelpooling . These operations assign the layer ’ s input signal ( feature map ) to the VQ partition region corresponding to the closest centroid in terms of the Euclidean distance , Contribution 2 : A hard VQ inference contains no information regarding the confidence of the VQ region selection , which is related to the distance from the input signal to the region boundary . In response , we develop a method for soft MAP inference of the VQ parameters based on the probability that the layer input belongs to a given VQ region . Switching from hard to soft VQ inference recovers several classical and powerful nonlinearities and provides an avenue to derive completely new ones . We illustrate by showing that the soft versions of ReLU and max-pooling are the sigmoid gated linear unit and softmax pooling , respectively . We also find a home for the sigmoid , hyperbolic tangent , and softmax in the framework as a new kind of DN layer where the MASO output is the VQ probability . Contribution 3 : We generalize hard and soft VQ to what we call β-VQ inference , where β ∈ ( 0 , 1 ) is a free and learnable parameter . This parameter interpolates the VQ from linear ( β → 0 ) , to probabilistic SMASO ( β = 0.5 ) , to deterministic MASO ( β → 1 ) . We show that the β-VQ version of the hard ReLU activation is the swish nonlinearity , which offers state-of-the-art performance in a range of computer vision tasks but was developed ad hoc through experimentation Ramachandran et al . ( 2017 ) . Contribution 4 : Seen through the MASO lens , current DNs solve a simplistic per-unit ( per-neuron ) , independent VQ optimization problem at each layer . In response , we extend the SMASO GMM to a factorial GMM that that supports jointly optimal VQ across all units in a layer . Since the factorial aspect of the new model would make naı̈ve VQ inference exponentially computationally complex , we develop a simple sufficient condition under which a we can achieve efficient , tractable , jointly optimal VQ inference . The condition is that the linear “ filters ” feeding into any nonlinearity should be orthogonal . We propose two simple strategies to learn approximately and truly orthogonal weights and show on three different datasets that both offer significant improvements in classification per- formance . Since orthogonalization can be applied to an arbitrary DN , this result and our theoretical understanding are of independent interest . This paper is organized as follows . After reviewing the theory of MASOs and VQ for DNs in Section 2 , we formulate the GMM-based extension to SMASOs in Section 3 . Section 4 develops the hybrid β-VQ inference with a special case study on the swish nonlinearity . Section 5 extends the SMASO to a factorial GMM and shows the power of DN orthogonalization . We wrap up in Section 6 with directions for future research . Proofs of the various results appear in several appendices in the Supplementary Material . 2 BACKGROUND ON MAX-AFFINE SPLINES AND DEEP NETWORKS . We first briefly review max-affine spline operators ( MASOs ) in the context of understanding the inner workings of DNs Balestriero & Baraniuk ( 2018a ; b ) . A MASO is an operator S [ A , B ] : RD → RK that maps an input vector of length D into an output vector of length K by concatenating K independent max-affine splines Magnani & Boyd ( 2009 ) ; Hannah & Dunson ( 2013 ) , with each spline formed from R piecewise affine and convex mappings . The MASO parameters consist of the “ slopes ” A ∈ RK×R×D and the “ offsets/biases ” B ∈ RK×R . See Appendix A for the precise definition . Given the input x ∈ RD and parameters A , B , a MASO produces the output z ∈ RK via [ z ] k = [ S [ A , B ] ( x ) ] k = maxr=1 , ... , R ( 〈 [ A ] k , r , · , x〉+ [ B ] k , r ) , ( 1 ) where [ z ] k denotes the kth dimension of z . The three subscripts of the slopes tensor [ A ] k , r , d correspond to output k , partition region r , and input signal index d. The two subscripts of the offsets/biases tensor [ B ] k , r correspond to output k and partition region r. An important consequence of ( 1 ) is that a MASO is completely determined by its slope and offset parameters without needing to specify the partition of the input space ( the “ knots ” when D = 1 ) . Indeed , solving ( 1 ) automatically computes an optimized partition of the input space RD that is equivalent to a vector quantization ( VQ ) Nasrabadi & King ( 1988 ) ; Gersho & Gray ( 2012 ) . We can make the VQ aspect explicit by rewriting ( 1 ) in terms of the Hard-VQ ( HVQ ) matrix TH ∈ RK×R . that contains K stacked one-hot row vectors , each with the one-hot position at index [ t ] k ∈ { 1 , . . . , R } corresponding to the arg max over r = 1 , . . . , R of ( 1 ) . Given the HVQ matrix , ( or equivalently , a region of the input space ) , the input-output mapping is affine and fully determined by [ z ] k = R∑ r=1 [ TH ] k , r ( 〈 [ A ] k , r , · , x〉+ [ B ] k , r ) . ( 2 ) We retrieve ( 1 ) from ( 2 ) by noting that [ t ] k = arg maxr=1 , ... , R ( 〈 [ A ] k , r , · , x〉+ [ B ] k , r ) . The key background result for this paper is that the layers of a very large class of DN are MASOs . Hence , such a DN is a composition of MASOs , where each layer MASO has as input the feature map z ( ` −1 ) ∈ RD ( ` −1 ) and produces z ( ` ) ∈ RD ( ` ) , with ` corresponding to the layer . Each MASO has thus specific parameters A ( ` ) , B ( ` ) . Theorem 1 . Any DN layer comprising a linear operator ( e.g. , fully connected or convolution ) composed with a convex and piecewise affine operator ( such as a ReLU , leaky-ReLU , or absolute value activation ; max/average/channel-pooling ; maxout ; all with or without skip connections ) is a MASO Balestriero & Baraniuk ( 2018a ; b ) . Appendix A provides the parameters A ( ` ) , B ( ` ) for the MASO corresponding to the ` th layer of any DN constructed from linear plus piecewise affine and convex components . Given this connection , we will identify z ( ` −1 ) above as the input ( feature map ) to the MASO DN layer and z ( ` ) as the output ( feature map ) . We also identify [ z ( ` ) ] k in ( 1 ) and ( 2 ) as the output of the kth unit ( aka neuron ) of the ` th layer . MASOs for higher-dimensional tensor inputs/outputs are easily developed by flattening .

Summary: This work extends the applicability of the spline theory of deep networks explored in previous works of Balestriero/ Baraniuk. The previous works setup DNs as layer-wise max-affine spline operators (MASOs) and recovers several non-linearities practically used as special cases of these MASOs. The previous works already recover RELU variants and some downsampling operators that the current submission characterizes as "hard" quantization.