Controlling the Complexity and Lipschitz Constant improves Polynomial Nets

1 INTRODUCTION . Recently , high-degree Polynomial Nets ( PNs ) have been demonstrating state-of-the-art performance in a range of challenging tasks like image generation ( Karras et al. , 2019 ; Chrysos and Panagakis , 2020 ) , image classification ( Wang et al. , 2018 ) , reinforcement learning ( Jayakumar et al. , 2020 ) , non-euclidean representation learning ( Chrysos et al. , 2020 ) and sequence models ( Su et al. , 2020 ) . In particular , in public benchmarks like the Face verification on MegaFace task1 ( KemelmacherShlizerman et al. , 2016 ) , Polynomial Nets are currently the top performing model . A major advantage of Polynomial Nets over traditional Neural Networks2 is that they are compatible with efficient Leveled Fully Homomorphic Encryption ( LFHE ) protocols ( Brakerski et al. , 2014 ) . Such protocols allow efficient computation on encrypted data , but they only support addition or multiplication operations i.e. , polynomials . This has prompted an effort to adapt neural networks by replacing typical activation functions with polynomial approximations ( Gilad-Bachrach et al. , 2016 ; Hesamifard et al. , 2018 ) . Polynomial Nets do not need any adaptation to work with LFHE . Without doubt , these arguments motivate further investigation about the inner-workings of Polynomial Nets . Surprisingly , little is known about the theoretical properties of such high-degree polynomial expansions , despite their success . Previous work on PNs ( Chrysos et al. , 2020 ; Chrysos and Panagakis , 2020 ) have focused on developing the foundational structure of the models as well as their training , but do not provide an analysis of their generalization ability or robustness to adversarial perturbations . In contrast , such type of results are readily available for traditional feed-forward Deep Neural Networks , in the form of high-probability generalization error bounds ( Neyshabur et al. , 2015 ; Bartlett et al. , 2017 ; Neyshabur et al. , 2017 ; Golowich et al. , 2018 ) or upper bounds on their Lipschitz constant ( Scaman and Virmaux , 2018 ; Fazlyab et al. , 2019 ; Latorre et al. , 2020 ) . Despite their similarity in the compositional structure , theoretical results for Deep Neural Networks2 do not apply to Polynomial Nets , as they are essentialy two non-overlapping classes of functions . Why are such results important ? First , they provide key theoretical quantities like the sample complexity of a hypothesis class : how many samples are required to succeed at learning in the PAC-framework . Second , they provide certified performance guarantees to adversarial perturbations 1https : //paperswithcode.com/sota/face-verification-on-megaface 2with non-polynomial activation functions . ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) via a worst-case analysis c.f . Scaman and Virmaux ( 2018 ) . Most importantly , the bounds themselves provide a principled way to regularize the hypothesis class and improve their accuracy or robustness . For example , Generalization and Lipschitz constant bounds of Deep Neural Networks that depend on the operator-norm of their weight matrices ( Bartlett et al. , 2017 ; Neyshabur et al. , 2017 ) have layed out the path for regularization schemes like spectral regularization ( Yoshida and Miyato , 2017 ; Miyato et al. , 2018 ) , Lipschitz-margin training ( Tsuzuku et al. , 2018 ) and Parseval Networks ( Cisse et al. , 2017 ) , to name a few . Indeed , such schemes have been observed in practice to improve the performance of Deep Neural Networks . Unfortunately , similar regularization schemes for Polynomial Nets do not exist due to the lack of analogous bounds . Hence , it is possible that PNs are not yet being used to their fullest potential . We believe that theoretical advances in their understanding might lead to more resilient and accurate models . In this work , we aim to fill the gap in the theoretical understanding of PNs . We summarize our main contributions as follows : Rademacher Complexity Bounds . We derive bounds on the Rademacher Complexity of the Coupled CP-decomposition model ( CCP ) and Nested Coupled CP-decomposition model ( NCP ) of PNs , under the assumption of a unit ` ∞-norm bound on the input ( Theorems 1 and 3 ) , a natural assumption in image-based applications . Analogous bounds for the ` 2-norm are also provided ( Appendices E.3 and F.3 ) . Such bounds lead to the first known generalization error bounds for this class of models . Lipschitz constant Bounds . To complement our understanding of the CCP and NCP models , we derive upper bounds on their ` ∞-Lipschitz constants ( Theorems 2 and 4 ) , which are directly related to their robustness against ` ∞-bounded adversarial perturbations , and provide formal guarantees . Analogous results hold for any ` p-norm ( Appendices E.4 and F.4 ) . Regularization schemes . We identify key quantities that simultaneously control both Rademacher Complexity and Lipschitz constant bounds that we previously derived , i.e. , the operator norms of the weight matrices in the Polynomial Nets . Hence , we propose to regularize the CCP and NCP models by constraining such operator norms . In doing so , our theoretical results indicate that both the generalization and the robustness to adversarial perturbations should improve . We propose a Projected Stochastic Gradient Descent scheme ( Algorithm 1 ) , enjoying the same per-iteration complexity as vanilla back-propagation in the ` ∞-norm case , and a variant that augments the base algorithm with adversarial traning ( Algorithm 2 ) . Experiments . We conduct experimentation in five widely-used datasets on image recognition and on dataset in audio recognition . The experimentation illustrates how the aforementioned regularization schemes impact the accuracy ( and the robust accuracy ) of both CCP and NCP models , outperforming alternative schemes such as Jacobian regularization and the L2 weight decay . Indeed , for a grid of regularization parameters we observe that there exists a sweet-spot for the regularization parameter which not only increases the test-accuracy of the model , but also its resilience to adversarial perturbations . Larger values of the regularization parameter also allow a trade-off between accuracy and robustness . The observation is consistent across all datasets and all adversarial attacks demonstrating the efficacy of the proposed regularization scheme . 2 RADEMACHER COMPLEXITY AND LIPSCHITZ CONSTANT BOUNDS FOR POLYNOMIAL NETS . Notation . The symbol ◦ denotes the Hadamard ( element-wise ) product , the symbol • is the facesplitting product , while the symbol ? denotes a convolutional operator . Matrices are denoted by uppercase letters e.g. , V . Due to the space constraints , a detailed notation is deferred to Appendix C. Assumption on the input distribution . Unless explicitly mentioned otherwise , we assume an ` ∞-norm unit bound on the input data i.e. , ‖z‖∞ ≤ 1 for any input z . This is the most common assumption in image-domain applications in contemporary deep learning , i.e. , each pixel takes values in [ −1 , 1 ] interval . Nevertheless , analogous results for ` 2-norm unit bound assumptions are presented in Appendices E.3 , E.4 , F.3 and F.4 . We now introduce the basic concepts that will be developed throughout the paper i.e. , the Rademacher Complexity of a class of functions ( Bartlett and Mendelson , 2002 ) and the Lipschitz constant . Definition 1 ( Empirical Rademacher Complexity ) . Let Z = { z1 , . . . , zn } ⊆ Rd and let { σj : j = 1 , . . . , n } be independent Rademacher random variables i.e. , taking values uniformly in { −1 , +1 } . Let F be a class of real-valued functions over Rd . The Empirical Rademacher complexity of F with respect to Z is defined as follows : RZ ( F ) : = Eσ [ sup f∈F 1 n n∑ j=1 σjf ( zj ) ] . Definition 2 ( Lipschitz constant ) . Given two normed spaces ( X , ‖ · ‖X ) and ( Y , ‖ · ‖Y ) , a function f : X → Y is called Lipschitz continuous with Lipschitz constant K ≥ 0 if for all x1 , x2 in X : ‖f ( x1 ) − f ( x2 ) ‖Y ≤ K‖x1 − x2‖X . 2.1 COUPLED CP-DECOMPOSITION OF POLYNOMIAL NETS ( CCP MODEL ) . The Coupled CP-Decomposition ( CCP ) model of PNs ( Chrysos et al. , 2020 ) leverages a coupled CP Tensor decomposition ( Kolda and Bader , 2009 ) to vastly reduce the parameters required to describe a high-degree polynomial , and allows its computation in a compositional fashion , much similar to a feed-forward pass through a traditional neural network . The CCP model was used in Chrysos and Panagakis ( 2020 ) to construct a generative model . CCP can be succintly defined as follows : f ( z ) = C ◦ki=1 Uiz , ( CCP ) where z ∈ Rd is the input data , f ( z ) ∈ Ro is the output of the model and Un ∈ Rm×d , C ∈ Ro×m are the learnable parameters , where m ∈ N is the hidden rank . In Fig . 1 we provide a schematic of the architecture , while in Appendix D.1 we include further details on the original CCP formulation ( and how to obtain our equivalent re-parametrization ) for the interested reader . In Theorem 1 we derive an upper bound on the complexity of CCP models with bounded ` ∞-operatornorms of the face-splitting product of the weight matrices . Its proof can be found in Appendix E.1 . For a given CCP model , we derive an upper bound on its ` ∞-Lipschitz constant in Theorem 2 and its proof is given in Appendix E.4.1 . Theorem 1 . Let Z = { z1 , . . . , zn } ⊆ Rd and suppose that ‖zj‖∞ ≤ 1 for all j = 1 , . . . , n. Let FkCCP : = { f ( z ) = 〈 c , ◦ki=1Uiz 〉 : ‖c‖1 ≤ µ , ∥∥•ki=1Ui∥∥∞ ≤ λ } . The Empirical Rademacher Complexity of CCPk ( k-degree CCP polynomials ) with respect to Z is bounded as : RZ ( FkCCP ) ≤ 2µλ √ 2k log ( d ) n . Proof sketch of Theorem 1 We now describe the core steps of the proof . For the interested reader , the complete and detailed proof steps are presented in Appendix E.1 . First , Hölder ’ s inequality is used to bound the Rademacher complexity as : RZ ( FkCCP ) = E sup f∈FkCCP 1 n 〈 c , n∑ j=1 [ σj◦ki=1 ( Uizj ) ] 〉 ≤ E sup f∈FkCCP 1 n ‖c‖1 ∥∥∥∥∥∥ n∑ j=1 [ σj◦ki=1 ( Uizj ) ] ∥∥∥∥∥∥ ∞ . ( 1 ) This shows why the factor ‖c‖1 ≤ µ appears in the final bound . Then , using the mixed product property ( Slyusar , 1999 ) and its extension to repeated Hadamard products ( Lemma 7 in Appendix C.3 ) , we can rewrite the summation in the right-hand-side of ( 1 ) as follows : n∑ j=1 σj◦ki=1 ( Uizj ) = n∑ j=1 σj •ki=1 ( Ui ) ∗ki=1 ( zj ) = •ki=1 ( Ui ) n∑ j=1 σj ∗ki=1 ( zj ) . This step can be seen as a linearization of the polynomial by lifting the problem to a higher dimensional space . We use this fact and the definition of the operator norm to further bound the term inside the ` ∞-norm in the right-hand-side of ( 1 ) . Such term is bounded as the product of the ` ∞-operator norm of •ki=1 ( Ui ) , and the ` ∞-norm of an expression involving the Rademacher variables σj and the vectors ∗ki=1 ( zj ) . Finally , an application of Massart ’ s Lemma ( Shalev-Shwartz and Ben-David ( 2014 ) , Lemma 26.8 ) leads to the final result . Theorem 2 . The Lipschitz constant ( with respect to the ` ∞-norm ) of the function defined in Eq . ( CCP ) , restricted to the set { z ∈ Rd : ‖z‖∞ ≤ 1 } is bounded as : Lip∞ ( f ) ≤ k‖C‖∞ k∏ i=1 ‖Ui‖∞ .

Summary: The paper introduces Rademacher Complexity bounds and Lipschitz constant bounds (both $\ell_2$ and $\ell_\infty$) for two families of polynomial networks. While prior theoretical results on complexity and Lipschitz constant bounds exist for standard neural networks that consist of linear layers and activation functions, these results do not directly apply to Polynomial networks due to their vastly different structures from standard neural networks.   On the practical side, the authors found that imposing $L_\infty$ norm constraint on the weight matrices of the polynomial networks is effective against adversarial perturbation. In particular, the paper focuses on robust image and audio recognition tasks and found out that imposing $L_\infty$ norm constraint via projected gradient descent is more effective than other commonly deployed regularization techniques (in standard networks) such as Jacobian regularization and weight decay.