Kernel Deformed Exponential Families for Sparse Continuous Attention

1 INTRODUCTION . Attention mechanisms take weighted averages of data representations ( Bahdanau et al. , 2015 ) , where the weights are a function of input objects . These are then used as inputs for prediction . Discrete attention 1 ) can not easily handle data where observations are irregularly spaced 2 ) attention maps may be scattered , lacking focus . Martins et al . ( 2020 ; 2021 ) extended attention to continuous settings , showing that attention densities maximize the regularized expectation of a function of the data location ( i.e . time ) . Special case solutions lead to exponential and deformed exponential families : the latter has sparse support . They form a continuous data representation and take expectations with respect to attention densities . Using measure theory to unify discrete and continuous approaches , they show transformer self-attention ( Vaswani et al. , 2017 ) is a special case of their formulation . Martins et al . ( 2020 ; 2021 ) explored unimodal attention densities : these only give high importance to one region of data . Farinhas et al . ( 2021 ) extended this to use multi-modal mixture of Gaussian attention densities . However 1 ) mixture of Gaussians do not lie in either the exponential or deformed exponential families , and are difficult to study in the context of Martins et al . ( 2020 ; 2021 ) 2 ) they have dense support . Sparse support can say that certain regions of data do not matter : a region of time has no effect on class probabilities , or a region of an image is not some object . We would like to use multimodal exponential and deformed exponential family attention densities , and understand how Farinhas et al . ( 2021 ) relates to the optimization problem of Martins et al . ( 2020 ; 2021 ) . This paper makes three contributions : 1 ) we introduce kernel deformed exponential families , a multimodal class of densities with sparse support and apply it along with the multimodal kernel exponential families ( Canu & Smola , 2006 ) to attention mechanisms . The latter have been used for density estimation , but not weighting data importance 2 ) we theoretically analyze normalization for both kernel exponential and deformed exponential families in terms of a base density and kernel , and show approximation properties for the latter 3 ) we apply them to real world datasets and show that kernel deformed exponential families learn flexible continuous attention densities with sparse support . Approximation properties for the kernel deformed are challenging : similar kernel exponential family results ( Sriperumbudur et al. , 2017 ) relied on standard exponential and logarithm properties to bound the difference of the log-partition functional at two functions : these do not hold for deformed analogues . We provide similar bounds via the functional mean value theorem along with bounding the Frechet derivative of the deformed log-partition functional . The paper is organized as follows : we review continuous attention ( Martins et al. , 2020 ; 2021 ) . We then describe how mixture of Gaussian attention densities , used in Farinhas et al . ( 2021 ) , solve a different optimization problem . We next describe kernel exponential families and give novel normalization condition relating the kernel growth to the base density ’ s tail decay . We then propose kernel deformed exponential families , which can have support over disjoint regions . We describe normalization and prove approximation capabilities . Next we describe use of these densities for continuous attention , including experiments where we show that the kernel deformed case learns multimodal attention densities with sparse support . We conclude with limitations and future work . 2 RELATED WORK . Attention Mechanisms closely related are Martins et al . ( 2020 ; 2021 ) ; Farinhas et al . ( 2021 ) ; Tsai et al . ( 2019 ) ; Shukla & Marlin ( 2021 ) . Martins et al . ( 2020 ; 2021 ) frame continuous attention as an expectation of a value function over the domain with respect to a density , where the density solves an optimization problem . They only used unimodal exponential and deformed exponential family densities : we extend this to the multimodal setting by leveraging kernel exponential families and proposing a deformed counterpart . Farinhas et al . ( 2021 ) proposed a multi-modal continuous attention mechanism via a mixture of Gaussians approach . We show that this solves a slightly different optimization problem from Martins et al . ( 2020 ; 2021 ) , and extend to two further general density classes . Shukla & Marlin ( 2021 ) provide an attention mechanism for irregularly sampled time series by use of a continuous-time kernel regression framework , but do not actually take an expectation of a data representation over time with respect to a continuous pdf , evaluating the kernel regression model at a fixed set of time points to obtain a discrete representation . This describes importance of data at a set of points rather than over regions . Other papers connect attention and kernels , but focus on the discrete attention setting ( Tsai et al. , 2019 ; Choromanski et al. , 2020 ) . Also relevant are temporal transformer papers , including Xu et al . ( 2019 ) ; Li et al . ( 2019 ; 2020 ) ; Song et al . ( 2018 ) . However none have continuous attention densities . Kernel Exponential Families Canu & Smola ( 2006 ) proposed kernel exponential families : Sriperumbudur et al . ( 2017 ) analyzed theory for density estimation . Wenliang et al . ( 2019 ) parametrized the kernel with a deep neural network . Other density estimation papers include Arbel & Gretton ( 2018 ) ; Dai et al . ( 2019 ) ; Sutherland et al . ( 2018 ) . We apply kernel exponential families as attention densities to weight a value function which represents the data , rather than to estimate the data density , and extend similar ideas to kernel deformed exponential families with sparse support . Wenliang et al . ( 2019 ) showed a condition for an unnormalized kernel exponential family density to have a finite normalizer . However , they used exponential power base densities . We instead relate kernel growth rates to the base density tail decay , allowing non-symmetric base densities . To summarize our theoretical contributions : 1 ) introducing kernel deformed exponential families with approximation and normalization analysis 2 ) improved kernel exponential family normalization results . 3 CONTINUOUS ATTENTION MECHANISMS . An attention mechanism involves : 1 ) the value function approximates a data representation . This may be the original data or a learned representation . 2 ) the attention density is chosen to be ’ similar ’ to another data representation , encoding it into a density 3 ) the context combines the two , taking an expectation of the value function with respect to the attention density . Formally , the context is c = ET∼p [ V ( T ) ] . ( 1 ) Here V ( t ) the value function approximates a data representation , T ∼ p ( t ) is the random variable or vector for locations ( temporal , spatial , etc ) , and p ( t ) is the attention density . To choose the attention density p , one takes a data representation f and finds p ’ similar ’ to f and thus to a data representation , but regularizing p. Martins et al . ( 2020 ; 2021 ) did this , providing a rigorous formulation of attention mechanisms . Given a probability space ( S , A , Q ) , letM1+ ( S ) be the set of densities with respect to Q . Assume that Q is dominated by a measure ν ( i.e . Lebesgue ) and that it has density q0 = dQdν with respect to ν . Let S ⊆ R D , F be a function class , and Ω : M1+ ( S ) → R be a lower semi-continuous , proper , strictly convex functional . Given f ∈ F , an attention density ( Martins et al. , 2020 ) p̂ : F → R≥0 solves p̂ [ f ] = arg max p∈M1+ ( S ) 〈p , f〉L2 ( Q ) − Ω ( p ) . ( 2 ) This maximizes regularized L2 similarity between p and a data representation f . If Ω ( p ) is the negative differential entropy , the attention density is Boltzmann Gibbs p̂ [ f ] ( t ) = exp ( f ( t ) −A ( f ) ) , ( 3 ) where A ( f ) ensures ∫ S p̂ [ f ] ( t ) dQ = 1 . If f ( t ) = θTφ ( t ) for parameters and statistics θ ∈ RM , φ ( t ) ∈ RM respectively , Eqn . 3 becomes an exponential family density . For f in a reproducing kernel Hilbert space H , it becomes a kernel exponential family density ( Canu & Smola , 2006 ) , which we propose to use as an alternative attention density . One desirable class would be heavy or thin tailed exponential family-like densities . In exponential families , the support , or non-negative region of the density , is controlled by the measure Q . Letting Ω ( p ) be the α-Tsallis negative entropy Ωα ( p ) ( Tsallis , 1988 ) , Ωα ( p ) = { 1 α ( α−1 ) ( ∫ S p ( t ) αdQ− 1 ) , α 6= 1 ; ∫ S p ( t ) log p ( t ) dQ , α = 1 , then p̂ [ f ] for f ( t ) = θTφ ( t ) lies in the deformed exponential family ( Tsallis , 1988 ; Naudts , 2004 ) p̂Ωα [ f ] ( t ) = exp2−α ( θ Tφ ( t ) −Aα ( f ) ) , ( 4 ) where Aα ( f ) again ensures normalization and the density uses the β-exponential expβ ( t ) = { [ 1 + ( 1− β ) t ] 1/ ( 1−β ) + , β 6= 1 ; exp ( t ) , β = 1 . ( 5 ) For β < 1 , Eqn . 5 and thus deformed exponential family densities for 1 < α ≤ 2 can return 0 values . Values α > 1 ( and thus β < 1 ) give thinner tails than the exponential family , while α < 1 gives fatter tails . Setting β = 0 is called sparsemax ( Martins & Astudillo , 2016 ) . In this paper , we assume 1 < α ≤ 2 , which is the sparse case studied in Martins et al . ( 2020 ) . We again propose to replace f ( t ) = θTφ ( t ) with f ∈ H , which leads to the novel kernel deformed exponential families . Computing Eqn . 1 ’ s context vector requires parametrizing V ( t ) . Martins et al . ( 2020 ) obtain a value function V : S → RD parametrized by B ∈ RD×N by applying regularized multivariate linear regression to estimate V ( t ; B ) = BΨ ( t ) , where Ψ = { ψn } Nn=1 is a set of basis functions . Let L be the number of observation locations ( times in a temporal setting ) , O be the observation dimension , andN be the number of basis functions . This involves regressing the observation matrix H ∈ RO×L on a matrix F ∈ RN×L of basis functions { ψn } Nn=1 evaluated at observation locations { tl } Ll=1 B∗ = arg min B ‖BF−H‖2F + λ‖B‖2F . ( 6 )

Summary: The paper studies the continuous attention mechanism using the kernel exponential family and its deformed variant. This is an extension of existing works based on finite-dimensional exponential families. The authors investigated some theoretical conditions such that the RKHS defines the probability density functions. Numerical experiments showed that the proposed method works efficiently for some datasets.