Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machines

Despite their ubiquity in core AI fields like natural language processing , the mechanics of deep attention-based neural networks like the “ Transformer ” model are not fully understood . In this article , we present a new perspective towards understanding how Transformers work . In particular , we show that the “ dot-product attention ” that is the core of the Transformer ’ s operation can be characterized as a kernel learning method on a pair of Banach spaces . In particular , the Transformer ’ s kernel is characterized as having an infinite feature dimension . Along the way we generalize the standard kernel learning problem to what we term a “ binary ” kernel learning problem , where data come from two input domains and a response is defined for every cross-domain pair . We prove a new representer theorem for these binary kernel machines with non-Mercer ( indefinite , asymmetric ) kernels ( implying that the functions learned are elements of reproducing kernel Banach spaces rather than Hilbert spaces ) , and also prove a new universal approximation theorem showing that the Transformer calculation can learn any binary non-Mercer reproducing kernel Banach space pair . We experiment with new kernels in Transformers , and obtain results that suggest the infinite dimensionality of the standard Transformer kernel is partially responsible for its performance . This paper ’ s results provide a new theoretical understanding of a very important but poorly understood model in modern machine learning . 1 INTRODUCTION . Since its proposal by Bahdanau et al . ( 2015 ) , so-called neural attention has become the backbone of many state-of-the-art deep learning models . This is true in particular in natural language processing ( NLP ) , where the Transformer model of Vaswani et al . ( 2017 ) has become ubiquitous . This ubiquity is such that much of the last few years ’ NLP breakthroughs have been due to developing new training regimes for Transformers ( Radford et al. , 2018 ; Devlin et al. , 2019 ; Yang et al. , 2019 ; Liu et al. , 2019 ; Wang et al. , 2019a ; Joshi et al. , 2020 ; Lan et al. , 2020 ; Brown et al. , 2020 , etc. ) . Like most modern deep neural networks , theoretical understanding of the Transformer has lagged behind the rate of Transformer-based performance improvements on AI tasks like NLP . Recently , several authors have noted Transformer operations ’ relationship to other , better-understood topics in deep learning theory , like the similarities between attention and convolution ( Ramachandran et al. , 2019 ; Cordonnier et al. , 2020 ) and the design of the residual blocks in multi-layer Transformers ( e.g. , Lu et al . ( 2019 ) ; see also the reordering of the main learned ( fully-connected or attentional ) operation , elementwise nonlinearity , and normalization in the original Transformer authors ’ official reference codebase ( Vaswani et al. , 2018 ) and in some more recent studies of deeper Transformers ( Wang et al. , 2019b ) to the “ pre-norm ” ordering of normalize , learned operation , nonlinearity , add residual ordering of modern ( “ v2 ” ) Resnets ( He et al. , 2016 ) ) . In this paper , we propose a new lens to understand the central component of the Transformer , its “ dot-product attention ” operation . In particular , we show dot-product attention can be characterized as a particular class of kernel method ( Schölkopf & Smola , 2002 ) . More specifically , it is a so-called indefinite and asymmetric kernel method , which refer to two separate generalizations of the classic class of kernels that does not require the classic assumptions of symmetry and positive ( semi- ) definiteness ( Ong et al. , 2004 ; Balcan et al. , 2008 ; Zhang et al. , 2009 ; Wu et al. , 2010 ; Loosli et al. , 2016 ; Oglic & Gärtner , 2018 ; 2019 , etc. ) . We in fact show in Theorem 2 below that dot-product attention can learn any asymmetric indefinite kernel . This insight has several interesting implications . Most immediately , it provides some theoretical justification for one of the more mysterious components of the Transformer model . It also potentially opens the door for the application of decades of classic kernel method theory towards understanding one of today ’ s most important neural network models , perhaps similarly to how tools from digital signal processing are widely used to study convolutional neural networks . We make a first step on this last point in this paper , proposing a generalization of prior kernel methods we call “ binary ” kernel machines , that learns how to predict distinct values for pairs of elements across two input sets , similar to an attention model . The remainder of this paper is organized as follows . Section 2 reviews the mathematical background of both Transformers and classic kernel methods . Section 3 presents the definition of kernel machines on reproducing kernel Banach spaces ( RKBS ’ s ) that we use to characterize Transformers . In particular we note that the Transformer can be described as having an infinite-dimensional feature space . Section 4 begins our theoretical results , explicitly describing the Transformer in terms of reproducing kernels , including explicit formulations of the Transformer ’ s kernel feature maps and its relation to prior kernels . Section 5 discusses Transformers as kernel learners , including a new representer theorem and a characterization of stochastic-gradient-descent-trained attention networks as approximate kernel learners . In Section 6 , we present empirical evidence that the infinite-dimensional character of the Transformer kernel may be somewhat responsible for the model ’ s effectiveness . Section 7 concludes and summarizes our work . 2 BACKGROUND AND RELATED WORK . 2.1 TRANSFORMER NEURAL NETWORK MODELS . The Transformer model ( Vaswani et al. , 2017 ) has become ubiquitous in many core AI applications like natural language processing . Here , we review its core components . Say we have two ordered sets of vectors , a set of “ source ” elements { s1 , s2 , . . . , sS } , sj ∈ Rds and a set of “ target ” elements { t1 , t2 , . . . , tT } , ti ∈ Rdt . In its most general form , the neural-network “ attention ” operation that forms the backbone of the Transformer model is to compute , for each ti , a ti-specific embedding of the source sequence { sj } Sj=1.1 The particular function used in the Transformer is the so-called “ scaled dot-product ” attention , which takes the form aij = ( WQti ) T ( WKsj ) √ d αij = exp ( aij ) ∑S j=1 exp ( aij ) t′i = S∑ j=1 αijW V sj ( 1 ) where W V , WK ∈ Rds×d , and WQ ∈ Rdt×d are learnable weight matrices , usually called the “ value , ” “ key , ” and “ query ” weight matrices , respectively . Usually multiple so-called “ attention heads ” with independent parameter matrices implement several parallel computations of ( 1 ) , with the Cartesian product ( vector concatenation ) of several d-dimensional head outputs forming the final output t′i . Usually the unnormalized aij ’ s are called attention scores or attention logits , and the normalized αij ’ s are called attention weights . In this paper , we restrict our focus to the dot-product formulation of attention shown in ( 1 ) . Several other alternative forms of attention that perform roughly the same function ( i.e. , mapping from Rds × Rdt to R ) have been proposed ( Bahdanau et al. , 2015 ; Luong et al. , 2015 ; Veličković et al. , 2018 ; Battaglia et al. , 2018 , etc . ) but the dot-product formulation of the Transformer is by far the most popular . 2.2 KERNEL METHODS AND GENERALIZATIONS OF KERNELS . Kernel methods ( Schölkopf & Smola , 2002 ; Steinwart & Christmann , 2008 , etc . ) are a classic and powerful class of machine learning methods . The key component of kernel methods are the namesake 1Often , the source and target sets are taken to be the same , si = ti ∀i . This instance of attention is called self attention . kernel functions , which allow the efficient mapping of input data from a low-dimensional data domain , where linear solutions to problems like classification or regression may not be possible , to a high- or infinite-dimensional embedding domain , where linear solutions can be found . Given two nonempty sets X and Y , a kernel function κ is a continuous function κ : X × Y → R. In the next few sections , we will review the classic symmetric and positive ( semi- ) definite , or Mercer , kernels , then discuss more general forms . 2.2.1 SYMMETRIC AND POSITIVE SEMIDEFINITE ( MERCER ) KERNELS . If X = Y , and for all xi , xj ∈ X = Y , a particular kernel κ has the properties symmetry : κ ( xi , xj ) = κ ( xj , xi ) ( 2a ) positive ( semi- ) definiteness : cTKc ≥ 0 ∀ c ∈ Rn ; i , j = 1 , . . . , n ; n ∈ N ( 2b ) whereK in ( 2b ) is the Gram matrix , defined as Kij = κ ( xi , xj ) , then κ is said to be a Mercer kernel . For Mercer kernels , it is well-known that , among other facts , ( i ) we can define a Hilbert space of functions on X , denotedHκ ( called the reproducing kernel Hilbert space , or RKHS , associated with the reproducing kernel κ ) , ( ii ) Hκ has for each x a ( continuous ) unique element δx called a point evaluation functional , with the property f ( x ) = δx ( f ) ∀f ∈ Hκ , ( iii ) κ has the so-called reproducing property , 〈f , κ ( x , · ) 〉Hκ = f ( x ) ∀f ∈ Hκ , where 〈· , ·〉Hκ is the inner product on Hκ , and ( iv ) we can define a “ feature map ” Φ : X → FH , where FH is another Hilbert space sometimes called the feature space , and κ ( x , y ) = 〈Φ ( x ) , Φ ( y ) 〉FH ( where 〈· , ·〉FH is the inner product associated with FH ) . This last point gives rise to the kernel trick for RKHS ’ s . From a machine learning and optimization perspective , kernels that are symmetric and positive ( semi- ) definite ( PSD ) are desirable because those properties guarantee that empirical-risk-minimization kernel learning problems like support vector machines ( SVMs ) , Gaussian processes , etc . are convex . Convexity gives appealing guarantees for the tractability of a learning problem and optimality of solutions . 2.2.2 LEARNING WITH NON-MERCER KERNELS . Learning methods with non-Mercer kernels , or kernels that relax the assumptions ( 2 ) , have been studied for some time . One line of work ( Lin & Lin , 2003 ; Ong et al. , 2004 ; Chen & Ye , 2008 ; Luss & D ’ aspremont , 2008 ; Alabdulmohsin et al. , 2015 ; Loosli et al. , 2016 ; Oglic & Gärtner , 2018 ; 2019 , etc . ) has focused on learning with symmetric but indefinite kernels , i.e. , kernels that do not satisfy ( 2b ) . Indefinite kernels have been identified as reproducing kernels for so-called reproducing kernel Kreı̆n spaces ( RKKS ’ s ) since Schwartz ( 1964 ) and Alpay ( 1991 ) . Replacing a Mercer kernel in a learning problem like an SVM with an indefinite kernel makes the optimization problem nonconvex in general ( as the kernel Gram matrixK is no longer always PSD ) . Some early work in learning with indefinite kernels tried to ameliorate this problem by modifying the spectrum of the Gram matrix such that it again becomes PSD ( e.g. , Graepel et al. , 1998 ; Roth et al. , 2003 ; Wu et al. , 2005 ) . More recently , Loosli et al . ( 2016 ) ; Oglic & Gärtner ( 2018 ) , among others , have proposed optimization procedures to learn in the RKKS directly . They report better performance on some learning problems when using indefinite kernels than either popular Mercer kernels or spectrally-modified indefinite kernels , suggesting that sacrificing convexity can empirically give a performance boost . This conclusion is of course reminiscent of the concurrent experience of deep neural networks , which are hard to optimize due to their high degree of non-convexity , yet give superior performance to many other methods . Another line of work has explored the application of kernel methods to learning in more general Banach spaces , i.e. , reproducing kernel Banach spaces ( RKBS ’ s ) ( Zhang et al. , 2009 ) . Various constructions to serve as the reproducing kernel for a Banach space ( replacing the inner product of an RKHS ) have been proposed , including semi-inner products ( Zhang et al. , 2009 ) , positive-definite bilinear forms via a Fourier transform construction ( Fasshauer et al. , 2015 ) , and others ( Song et al. , 2013 ; Georgiev et al. , 2014 , etc. ) . In this work , we consider RKBS ’ s whose kernels may be neither symmetric nor PSD . A definition of these spaces is presented next .

Summary: In this paper, the authors treat a particular Transformer, "dot-product attention", as  an RKBS kernel called "exponentiated query-key kernel". The explicit form of feature maps and Bach space are given. Moreover, authors term a binary kernel learning problems within the framework of regularized empirical risk minimization. The problem and the correponding representer theorem is new due to its extension to Banach space. A new approximation theorem is also proved and  some experiements are done.