Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machines

Despite their ubiquity in core AI fields like natural language processing , the mechanics of deep attention-based neural networks like the “ Transformer ” model are not fully understood . In this article , we present a new perspective towards understanding how Transformers work . In particular , we show that the “ dot-product attention ” that is the core of the Transformer ’ s operation can be characterized as a kernel learning method on a pair of Banach spaces . In particular , the Transformer ’ s kernel is characterized as having an infinite feature dimension . Along the way we generalize the standard kernel learning problem to what we term a “ binary ” kernel learning problem , where data come from two input domains and a response is defined for every cross-domain pair . We prove a new representer theorem for these binary kernel machines with non-Mercer ( indefinite , asymmetric ) kernels ( implying that the functions learned are elements of reproducing kernel Banach spaces rather than Hilbert spaces ) , and also prove a new universal approximation theorem showing that the Transformer calculation can learn any binary non-Mercer reproducing kernel Banach space pair . We experiment with new kernels in Transformers , and obtain results that suggest the infinite dimensionality of the standard Transformer kernel is partially responsible for its performance . This paper ’ s results provide a new theoretical understanding of a very important but poorly understood model in modern machine learning . 1 INTRODUCTION . Since its proposal by Bahdanau et al . ( 2015 ) , so-called neural attention has become the backbone of many state-of-the-art deep learning models . This is true in particular in natural language processing ( NLP ) , where the Transformer model of Vaswani et al . ( 2017 ) has become ubiquitous . This ubiquity is such that much of the last few years ’ NLP breakthroughs have been due to developing new training regimes for Transformers ( Radford et al. , 2018 ; Devlin et al. , 2019 ; Yang et al. , 2019 ; Liu et al. , 2019 ; Wang et al. , 2019a ; Joshi et al. , 2020 ; Lan et al. , 2020 ; Brown et al. , 2020 , etc. ) . Like most modern deep neural networks , theoretical understanding of the Transformer has lagged behind the rate of Transformer-based performance improvements on AI tasks like NLP . Recently , several authors have noted Transformer operations ’ relationship to other , better-understood topics in deep learning theory , like the similarities between attention and convolution ( Ramachandran et al. , 2019 ; Cordonnier et al. , 2020 ) and the design of the residual blocks in multi-layer Transformers ( e.g. , Lu et al . ( 2019 ) ; see also the reordering of the main learned ( fully-connected or attentional ) operation , elementwise nonlinearity , and normalization in the original Transformer authors ’ official reference codebase ( Vaswani et al. , 2018 ) and in some more recent studies of deeper Transformers ( Wang et al. , 2019b ) to the “ pre-norm ” ordering of normalize , learned operation , nonlinearity , add residual ordering of modern ( “ v2 ” ) Resnets ( He et al. , 2016 ) ) . In this paper , we propose a new lens to understand the central component of the Transformer , its “ dot-product attention ” operation . In particular , we show dot-product attention can be characterized as a particular class of kernel method ( Schölkopf & Smola , 2002 ) . More specifically , it is a so-called indefinite and asymmetric kernel method , which refer to two separate generalizations of the classic class of kernels that does not require the classic assumptions of symmetry and positive ( semi- ) definiteness ( Ong et al. , 2004 ; Balcan et al. , 2008 ; Zhang et al. , 2009 ; Wu et al. , 2010 ; Loosli et al. , 2016 ; Oglic & Gärtner , 2018 ; 2019 , etc. ) . We in fact show in Theorem 2 below that dot-product attention can learn any asymmetric indefinite kernel . This insight has several interesting implications . Most immediately , it provides some theoretical justification for one of the more mysterious components of the Transformer model . It also potentially opens the door for the application of decades of classic kernel method theory towards understanding one of today ’ s most important neural network models , perhaps similarly to how tools from digital signal processing are widely used to study convolutional neural networks . We make a first step on this last point in this paper , proposing a generalization of prior kernel methods we call “ binary ” kernel machines , that learns how to predict distinct values for pairs of elements across two input sets , similar to an attention model . The remainder of this paper is organized as follows . Section 2 reviews the mathematical background of both Transformers and classic kernel methods . Section 3 presents the definition of kernel machines on reproducing kernel Banach spaces ( RKBS ’ s ) that we use to characterize Transformers . In particular we note that the Transformer can be described as having an infinite-dimensional feature space . Section 4 begins our theoretical results , explicitly describing the Transformer in terms of reproducing kernels , including explicit formulations of the Transformer ’ s kernel feature maps and its relation to prior kernels . Section 5 discusses Transformers as kernel learners , including a new representer theorem and a characterization of stochastic-gradient-descent-trained attention networks as approximate kernel learners . In Section 6 , we present empirical evidence that the infinite-dimensional character of the Transformer kernel may be somewhat responsible for the model ’ s effectiveness . Section 7 concludes and summarizes our work . 2 BACKGROUND AND RELATED WORK . 2.1 TRANSFORMER NEURAL NETWORK MODELS . The Transformer model ( Vaswani et al. , 2017 ) has become ubiquitous in many core AI applications like natural language processing . Here , we review its core components . Say we have two ordered sets of vectors , a set of “ source ” elements { s1 , s2 , . . . , sS } , sj ∈ Rds and a set of “ target ” elements { t1 , t2 , . . . , tT } , ti ∈ Rdt . In its most general form , the neural-network “ attention ” operation that forms the backbone of the Transformer model is to compute , for each ti , a ti-specific embedding of the source sequence { sj } Sj=1.1 The particular function used in the Transformer is the so-called “ scaled dot-product ” attention , which takes the form aij = ( WQti ) T ( WKsj ) √ d αij = exp ( aij ) ∑S j=1 exp ( aij ) t′i = S∑ j=1 αijW V sj ( 1 ) where W V , WK ∈ Rds×d , and WQ ∈ Rdt×d are learnable weight matrices , usually called the “ value , ” “ key , ” and “ query ” weight matrices , respectively . Usually multiple so-called “ attention heads ” with independent parameter matrices implement several parallel computations of ( 1 ) , with the Cartesian product ( vector concatenation ) of several d-dimensional head outputs forming the final output t′i . Usually the unnormalized aij ’ s are called attention scores or attention logits , and the normalized αij ’ s are called attention weights . In this paper , we restrict our focus to the dot-product formulation of attention shown in ( 1 ) . Several other alternative forms of attention that perform roughly the same function ( i.e. , mapping from Rds × Rdt to R ) have been proposed ( Bahdanau et al. , 2015 ; Luong et al. , 2015 ; Veličković et al. , 2018 ; Battaglia et al. , 2018 , etc . ) but the dot-product formulation of the Transformer is by far the most popular . 2.2 KERNEL METHODS AND GENERALIZATIONS OF KERNELS . Kernel methods ( Schölkopf & Smola , 2002 ; Steinwart & Christmann , 2008 , etc . ) are a classic and powerful class of machine learning methods . The key component of kernel methods are the namesake 1Often , the source and target sets are taken to be the same , si = ti ∀i . This instance of attention is called self attention . kernel functions , which allow the efficient mapping of input data from a low-dimensional data domain , where linear solutions to problems like classification or regression may not be possible , to a high- or infinite-dimensional embedding domain , where linear solutions can be found . Given two nonempty sets X and Y , a kernel function κ is a continuous function κ : X × Y → R. In the next few sections , we will review the classic symmetric and positive ( semi- ) definite , or Mercer , kernels , then discuss more general forms . 2.2.1 SYMMETRIC AND POSITIVE SEMIDEFINITE ( MERCER ) KERNELS . If X = Y , and for all xi , xj ∈ X = Y , a particular kernel κ has the properties symmetry : κ ( xi , xj ) = κ ( xj , xi ) ( 2a ) positive ( semi- ) definiteness : cTKc ≥ 0 ∀ c ∈ Rn ; i , j = 1 , . . . , n ; n ∈ N ( 2b ) whereK in ( 2b ) is the Gram matrix , defined as Kij = κ ( xi , xj ) , then κ is said to be a Mercer kernel . For Mercer kernels , it is well-known that , among other facts , ( i ) we can define a Hilbert space of functions on X , denotedHκ ( called the reproducing kernel Hilbert space , or RKHS , associated with the reproducing kernel κ ) , ( ii ) Hκ has for each x a ( continuous ) unique element δx called a point evaluation functional , with the property f ( x ) = δx ( f ) ∀f ∈ Hκ , ( iii ) κ has the so-called reproducing property , 〈f , κ ( x , · ) 〉Hκ = f ( x ) ∀f ∈ Hκ , where 〈· , ·〉Hκ is the inner product on Hκ , and ( iv ) we can define a “ feature map ” Φ : X → FH , where FH is another Hilbert space sometimes called the feature space , and κ ( x , y ) = 〈Φ ( x ) , Φ ( y ) 〉FH ( where 〈· , ·〉FH is the inner product associated with FH ) . This last point gives rise to the kernel trick for RKHS ’ s . From a machine learning and optimization perspective , kernels that are symmetric and positive ( semi- ) definite ( PSD ) are desirable because those properties guarantee that empirical-risk-minimization kernel learning problems like support vector machines ( SVMs ) , Gaussian processes , etc . are convex . Convexity gives appealing guarantees for the tractability of a learning problem and optimality of solutions . 2.2.2 LEARNING WITH NON-MERCER KERNELS . Learning methods with non-Mercer kernels , or kernels that relax the assumptions ( 2 ) , have been studied for some time . One line of work ( Lin & Lin , 2003 ; Ong et al. , 2004 ; Chen & Ye , 2008 ; Luss & D ’ aspremont , 2008 ; Alabdulmohsin et al. , 2015 ; Loosli et al. , 2016 ; Oglic & Gärtner , 2018 ; 2019 , etc . ) has focused on learning with symmetric but indefinite kernels , i.e. , kernels that do not satisfy ( 2b ) . Indefinite kernels have been identified as reproducing kernels for so-called reproducing kernel Kreı̆n spaces ( RKKS ’ s ) since Schwartz ( 1964 ) and Alpay ( 1991 ) . Replacing a Mercer kernel in a learning problem like an SVM with an indefinite kernel makes the optimization problem nonconvex in general ( as the kernel Gram matrixK is no longer always PSD ) . Some early work in learning with indefinite kernels tried to ameliorate this problem by modifying the spectrum of the Gram matrix such that it again becomes PSD ( e.g. , Graepel et al. , 1998 ; Roth et al. , 2003 ; Wu et al. , 2005 ) . More recently , Loosli et al . ( 2016 ) ; Oglic & Gärtner ( 2018 ) , among others , have proposed optimization procedures to learn in the RKKS directly . They report better performance on some learning problems when using indefinite kernels than either popular Mercer kernels or spectrally-modified indefinite kernels , suggesting that sacrificing convexity can empirically give a performance boost . This conclusion is of course reminiscent of the concurrent experience of deep neural networks , which are hard to optimize due to their high degree of non-convexity , yet give superior performance to many other methods . Another line of work has explored the application of kernel methods to learning in more general Banach spaces , i.e. , reproducing kernel Banach spaces ( RKBS ’ s ) ( Zhang et al. , 2009 ) . Various constructions to serve as the reproducing kernel for a Banach space ( replacing the inner product of an RKHS ) have been proposed , including semi-inner products ( Zhang et al. , 2009 ) , positive-definite bilinear forms via a Fourier transform construction ( Fasshauer et al. , 2015 ) , and others ( Song et al. , 2013 ; Georgiev et al. , 2014 , etc. ) . In this work , we consider RKBS ’ s whose kernels may be neither symmetric nor PSD . A definition of these spaces is presented next .

Summary: The paper aims at providing a mathematical structure for explaining the mechanism behind the attention block characteristic to transformers. The focus of the paper is on the scaled dot-product attention, reviewed in Eq. (1). In my understanding, the whole mechanism can be seen as an instance of the set kernel. The inputs are bags of items, where an item is denoted with $s_j$. The items are embedded into some feature space via matrix multiplication $W^V s_j$. The set kernel representation of a bag is obtained by weighted averaging of item embeddings, where the item-specific weight is the output of an exponential family model. The latter model is obtained by combining embeddings of items and corresponding context vectors, denoted with $t_i$ (see Eq. 1 for more details).