Monotonic Kronecker-Factored Lattice

1 INTRODUCTION . Many machine learning problems have other requirements in addition to accuracy , such as efficiency , storage , and interpretability . For example , the ability to learn flexible monotonic functions at scale is useful in practice because machine learning practitioners often know apriori which input features positively or negatively relate to the output and can incorporate these hints as inductive bias in the training to further regularize the model ( Abu-Mostafa , 1993 ) and guarantee its expected behavior on unseen examples . It is , however , computationally challenging to learn such functions efficiently . In this paper , we extend the work of interpretable monotonic lattice regression ( Gupta et al. , 2016 ) to significantly reduce computational and storage costs without compromising accuracy . While a linear model with nonnegative coefficients is able to learn simple monotonic functions , its function class is restricted . Prior works proposed non-linear methods ( Sill , 1997 ; Dugas et al. , 2009 ; Daniels & Velikova , 2010 ; Qu & Hu , 2011 ) to learn more flexible monotonic functions , but they are shown to work only in limited settings of small datasets and low dimensional feature spaces . Monotonic lattice regression ( Gupta et al. , 2016 ) , an extension of lattice regression ( Garcia et al. , 2012 ) , learns an interpolated look-up table with linear inequality constraints that impose monotonicity . This has been demonstrated to work with millions of training examples and achieve competitive accuracy against , for example , random forests ; however , because the number of model parameters scales exponentially in the number of input features , it is difficult to apply such models in high dimensional feature space settings . To overcome this limitation , ( Canini et al. , 2016 ) incorporated ensemble learning to combine many tiny lattice models , each capturing non-linear interactions among a small random subset of features . This paper proposes Kronecker-Factored Lattice ( KFL ) , a novel reparameterization of monotonic lattice regression via Kronecker product to achieve significant parameter efficiency and simultaneously provide guaranteed monotonicity of the learned model with respect to a user-prescribed set of input ∗Equal contribution . features for user trust . Both inference and storage costs scale linearly in the number of features . Hence , it potentially allows for more features to non-linearly interact . Kronecker factorization has been applied to a wide variety of problems including optimization ( Martens & Grosse , 2015 ; George et al. , 2018 ; Osawa et al. , 2019 ) , convolutional neural networks ( Zhang et al. , 2015 ) , and recurrent neural networks ( Jose et al. , 2018 ) , but , to the best of our knowledge , has not yet been explored to learn flexible monotonic functions . The main contributions of this paper are : ( 1 ) reparameterization of monotonic lattice regression to achieve linear evaluation time and storage of the resulting subclass of models , ( 2 ) proving sufficient and necessary conditions for a KFL model to be monotonic as well as convex and nonnegative , ( 3 ) showing how the conditions for monotonicity can be efficiently imposed during training , ( 4 ) characterizing the values of M for which the capacity of the function class of an ensemble of M base KFL models strictly increases and showing that with a large enough M an ensemble can express every multilinear interpolated lattice function , and ( 5 ) providing experimental results on both public and proprietary datasets that demonstrate that KFL has accuracy and evaluation speeds comparable to or better than other lattice methods while requiring less training time and fewer parameters . 2 NOTATION AND OVERVIEW OF MONOTONIC LATTICE REGRESSION . For n ∈ N we denote by [ n ] the set { 1 , 2 , . . . , n } . For x ∈ RD and d ∈ [ D ] , let x [ d ] denote the dth entry of x . We use ed , n to denote the one-hot vector in { 0 , 1 } n where ed , n [ j ] = 1 if j = d and ed , n [ j ] = 0 , otherwise . When n is clear from the context we write ed . For two real vectors v , w of the same length we use v · w to denote their dot product . We write v ≤ w and v ≥ w if the respective inequality holds entrywise . We denote by 0 the zero vector in a real vector space whose dimension is clear from the context . For two sets S and T we use S × T to denote their Cartesian product and S \ T to denote the set of elements in S but not in T . We denote by |S| the cardinality of S. For w ∈ RV , we use fpwl ( x ; w ) to denote the 1-dimensional continuous piecewise linear function fpwl ( · ; w ) : [ 0 , V − 1 ] → R whose graph has V−1 linear segments , where for i = 1 , . . . , V−1 , the ith segment connects the points ( i − 1 , w [ i ] ) with ( i , w [ i + 1 ] ) . See Figure 1 for an example . Observe that for any α , β ∈ R and vectors w , v ∈ RV , we have fpwl ( · ; αw + βv ) = αfpwl ( · ; w ) + βfpwl ( · ; v ) . Lattice models , proposed in Garcia et al . ( 2012 ) , are interpolated look-up tables such that the function parameters are the function values sampled on a regular grid . They have been shown to exhibit efficient training procedures such that the resulting model is guaranteed to satisfy various types of shape constraints , including monotonicity and convexity ( Gupta et al. , 2016 ; 2018 ; Cotter et al. , 2019 ) , without severely restricting the model class . See Figure 2 for an example . More precisely , a D-dimensional lattice of size V∈ND consists of a regular D-dimensional grid of look-up table verticesMV = { 0 , 1 , . . . , V [ 1 ] −1 } × . . . × { 0 , 1 , . . . , V [ D ] −1 } . Thus , V [ d ] is the number of vertices in the lattice in the dth dimension , and the grid has ∏D d=1 V [ d ] vertices . We denote by DV ∈ RD the domain of the lattice model given by DV = [ 0 , V [ 1 ] − 1 ] × . . . × [ 0 , V [ D ] − 1 ] . Typically , each feature d is `` calibrated '' so that its value lies in [ 0 , V [ d ] −1 ] , using the piecewise-linear calibration technique of ( Gupta et al. , 2016 ) ( see also Section 3.5 ) . The lattice parameters are a vector θ ∈ RMV , where RMV denotes the set of vectors of length |MV | with entries indexed byMV : for each vertex v ∈ MV , there is a corresponding parameter we denote by θv . The lattice model or function is obtained by interpolating the parameters using one of many possible interpolation schemes specified by an interpolation kernel Φ : DV → RMV . Given Φ , the resulting lattice model f : Dv → R is defined by f ( x ; θ ) = θ ·Φ ( x ) = ∑ v∈MV θv ( Φ ( x ) ) v , ( 1 ) where ( Φ ( x ) ) v denotes the entry of Φ ( x ) with index v. Typically , one requires that for all u , v ∈ MV , ( Φ ( v ) ) u = 1 if u = v , and 0 otherwise , so that f ( v ) = θv . In Gupta et al . ( 2016 ) the authors present two interpolation schemes termed ’ multilinear ’ and ’ simplex ’ . We denote the corresponding kernels by ΦmultilinearV and Φ simplex V and the resulting lattice models by fmll ( · ; θ ) and fsl ( · ; θ ) , respectively . We give the definition of ΦmultilinearV here since we make use of it later , but refer the reader to ( Gupta et al. , 2016 ) for the definition of simplex interpolation . ( ΦmultilinearV ( x ) ) v = ∏ j∈ [ D ] fpwl ( x [ j ] ; ev [ j ] +1 , V [ j ] ) , v ∈MV ( 2 ) For both multilinear and simplex interpolated lattice functions , the resulting function is increasing ( resp . decreasing ) in a given direction if and only if the function is increasing ( resp . decreasing ) on the grid ’ s vertices in that direction ( Gupta et al. , 2016 , Lemmas 1 , 3 ) . As a result , training a lattice model with these interpolation schemes to respect the monotonicity constraint can be done by solving a constrained optimization problem with linear inequality constraints ( Gupta et al. , 2016 ) . Evaluating the lattice function at a single point can be done inO ( 2D ) time for multilinear interpolation and in O ( D logD ) time for simplex interpolation ( Gupta et al. , 2016 ) . Both interpolation schemes require O ( ∏ d V [ d ] ) space for storing the parameters ( assuming a constant number of bits per parameter ) . In the next section we present a sub-class of the set of multlinear-interpolation lattice functions , which we term KFL ( V ) . Each function in this class can be evaluated in O ( D ) time and requires only O ( ∑ d V [ d ] ) space for its parameters . Moreover , like the multilinear and simplex interpolated lattice function classes , there is a necessary and sufficient condition for a function in KFL ( V ) to be monotonic in a subset of its inputs , and this condition can be efficiently checked . As a result , one can train models in KFL ( V ) that are guaranteed to be monotonic in a prescribed set of features . 3 KRONECKER-FACTORED LATTICE . The Kronecker product of two matrices A = ( ai , j ) ∈ Rm×n and B = ( bi , j ) ∈ Rp×q is defined as the mp× nq real block matrix given by : A⊗B = a11B · · · a1nB ... . . . ... am1B · · · amnB . ( 3 ) The Kronecker product is associative and generally non-commutative ; however , A⊗B and B⊗A are permutation equivalent . We extend the definition of Kronecker product to vectors by regarding them as matrices with a single row or column . For s vectors vi ∈ Rni , we use the notation ⊗si=1vi = v1 ⊗ . . .⊗ vs. Let I = { 0 , . . . , n1 − 1 } × . . .× { 0 , . . . , ns − 1 } . We index the entries of ⊗si=1vi by I , so that the entry with index ( i1 , . . . , is ) ∈ I is ∏s j=1 vj [ ij + 1 ] . The Kronecker product satisfies the mixed-product property : for 4 matrices A ∈ Rm×n , B ∈ Rp×q , C ∈ Rn×r , D ∈ Rq×s it holds that ( A ⊗B ) ( C ⊗D ) = ( AC ) ⊗ ( BD ) , where for two matrices M , N , we denote their conventional product by MN . Consequently for vectors a , c ∈ Rn and b , d ∈ Rq it holds that ( a⊗ b ) · ( c⊗ d ) = ( a · c ) ⊗ ( b · d ) = ( a · c ) ( b · d ) , ( 4 ) where the rightmost equality follows since Kronecker product of scalars reduces to a simple product . See ( Loan , 2000 ) for more details on Kronecker products . We are now ready to define the class KFL ( V ) . Let V ∈ ND and fix a lattice of size V . The class KFL ( V ) is the subclass of all multilinear interpolation lattice functions whose parameters are Kronecker products of D factors . Precisely , KFL ( V ) = { fmll ( · ; ⊗Dd=1wd ) : ∀d wd ∈ RV [ d ] } . We reparameterize the functions in KFL and use fkfl ( · ; w1 , . . . , wd ) = fmll ( · ; ⊗Dd=1wd ) to denote the function with parameters { wd } d. The following proposition shows that functions in KFL ( V ) are products of 1-dimensional piecewise linear functions . It essentially follows from the fact that ΦmultilinearV ( x ) can be expressed as a Kronecker product of D vectors . Proposition 1 . Let V ∈ ND . For all w1 ∈ RV [ 1 ] , . . . , wD ∈ RV [ D ] fkfl ( x ; w1 , . . . , wD ) = ∏ d fpwl ( x [ d ] ; wd ) , for all x ∈ DV . ( 5 ) Proof of Proposition 1 . For v ∈ N , let Ψv : R → Rv be given by Ψv ( x ) = [ fpwl ( x ; e1 , v ) fpwl ( x ; e2 , v ) . . . fpwl ( x ; ev , v ) ] . Let w1 , . . . , wD be vectors satisfying the conditions in the proposition . By the definition of KFL ( V ) , we have fkfl ( · ; w1 , . . . , wD ) = fmll ( · ; ⊗Dd=1wd ) . Fix Φ = ΦmultilinearV . It follows from ( 2 ) that for all x ∈ DV we have Φ ( x ) = ⊗Dd=1ΨV [ d ] ( x [ d ] ) . Therefore using ( 4 ) we get , for all x ∈ Dv , fkfl ( x ; w1 , . . . , wD ) = fmll ( x ; ⊗Dd=1wd ) = ( ⊗Dd=1wd ) · ( ⊗Dd=1ΨV [ d ] ( x [ d ] ) ) = D∏ d=1 ( wd ·ΨV [ d ] ( x [ d ] ) ) . ( 6 ) Now , for each d ∈ [ D ] we have wd ·ΨV [ d ] ( x [ d ] ) = ∑ i∈ [ V [ d ] ] wd [ i ] fpwl ( x [ d ] ; ei , V [ d ] ) = fpwl ( x [ d ] ; ∑ i wd [ i ] ei , V [ d ] ) = fpwl ( x [ d ] ; wd ) . ( 7 ) Substituting ( 7 ) into ( 6 ) , we obtain ( 5 ) . It follows from Proposition 1 that evaluating a function in KFL ( V ) requires evaluating D piecewise linear functions with uniformly spaced knots and computing the product . This can be done in time O ( D ) . The storage complexity is O ( # parameters ) = O ( ∑ i V [ i ] ) .

Summary: In this paper, the authors study statistical models based on the Kronecker-factored lattice (KFL) and investigated the condition that the monotonicity holds. The KFL provides efficient and flexible modeling for monotonic shape constraints. Furthermore, the ensemble of KFL is proposed, and its approximation ability is theoretically investigated. Some numerical experiments indicate that the KFL trains faster with fewer parameters with comparable prediction accuracy to existing methods.