Kernel Deformed Exponential Families for Sparse Continuous Attention

1 INTRODUCTION . Attention mechanisms take weighted averages of data representations ( Bahdanau et al. , 2015 ) , where the weights are a function of input objects . These are then used as inputs for prediction . Discrete attention 1 ) can not easily handle data where observations are irregularly spaced 2 ) attention maps may be scattered , lacking focus . Martins et al . ( 2020 ; 2021 ) extended attention to continuous settings , showing that attention densities maximize the regularized expectation of a function of the data location ( i.e . time ) . Special case solutions lead to exponential and deformed exponential families : the latter has sparse support . They form a continuous data representation and take expectations with respect to attention densities . Using measure theory to unify discrete and continuous approaches , they show transformer self-attention ( Vaswani et al. , 2017 ) is a special case of their formulation . Martins et al . ( 2020 ; 2021 ) explored unimodal attention densities : these only give high importance to one region of data . Farinhas et al . ( 2021 ) extended this to use multi-modal mixture of Gaussian attention densities . However 1 ) mixture of Gaussians do not lie in either the exponential or deformed exponential families , and are difficult to study in the context of Martins et al . ( 2020 ; 2021 ) 2 ) they have dense support . Sparse support can say that certain regions of data do not matter : a region of time has no effect on class probabilities , or a region of an image is not some object . We would like to use multimodal exponential and deformed exponential family attention densities , and understand how Farinhas et al . ( 2021 ) relates to the optimization problem of Martins et al . ( 2020 ; 2021 ) . This paper makes three contributions : 1 ) we introduce kernel deformed exponential families , a multimodal class of densities with sparse support and apply it along with the multimodal kernel exponential families ( Canu & Smola , 2006 ) to attention mechanisms . The latter have been used for density estimation , but not weighting data importance 2 ) we theoretically analyze normalization for both kernel exponential and deformed exponential families in terms of a base density and kernel , and show approximation properties for the latter 3 ) we apply them to real world datasets and show that kernel deformed exponential families learn flexible continuous attention densities with sparse support . Approximation properties for the kernel deformed are challenging : similar kernel exponential family results ( Sriperumbudur et al. , 2017 ) relied on standard exponential and logarithm properties to bound the difference of the log-partition functional at two functions : these do not hold for deformed analogues . We provide similar bounds via the functional mean value theorem along with bounding the Frechet derivative of the deformed log-partition functional . The paper is organized as follows : we review continuous attention ( Martins et al. , 2020 ; 2021 ) . We then describe how mixture of Gaussian attention densities , used in Farinhas et al . ( 2021 ) , solve a different optimization problem . We next describe kernel exponential families and give novel normalization condition relating the kernel growth to the base density ’ s tail decay . We then propose kernel deformed exponential families , which can have support over disjoint regions . We describe normalization and prove approximation capabilities . Next we describe use of these densities for continuous attention , including experiments where we show that the kernel deformed case learns multimodal attention densities with sparse support . We conclude with limitations and future work . 2 RELATED WORK . Attention Mechanisms closely related are Martins et al . ( 2020 ; 2021 ) ; Farinhas et al . ( 2021 ) ; Tsai et al . ( 2019 ) ; Shukla & Marlin ( 2021 ) . Martins et al . ( 2020 ; 2021 ) frame continuous attention as an expectation of a value function over the domain with respect to a density , where the density solves an optimization problem . They only used unimodal exponential and deformed exponential family densities : we extend this to the multimodal setting by leveraging kernel exponential families and proposing a deformed counterpart . Farinhas et al . ( 2021 ) proposed a multi-modal continuous attention mechanism via a mixture of Gaussians approach . We show that this solves a slightly different optimization problem from Martins et al . ( 2020 ; 2021 ) , and extend to two further general density classes . Shukla & Marlin ( 2021 ) provide an attention mechanism for irregularly sampled time series by use of a continuous-time kernel regression framework , but do not actually take an expectation of a data representation over time with respect to a continuous pdf , evaluating the kernel regression model at a fixed set of time points to obtain a discrete representation . This describes importance of data at a set of points rather than over regions . Other papers connect attention and kernels , but focus on the discrete attention setting ( Tsai et al. , 2019 ; Choromanski et al. , 2020 ) . Also relevant are temporal transformer papers , including Xu et al . ( 2019 ) ; Li et al . ( 2019 ; 2020 ) ; Song et al . ( 2018 ) . However none have continuous attention densities . Kernel Exponential Families Canu & Smola ( 2006 ) proposed kernel exponential families : Sriperumbudur et al . ( 2017 ) analyzed theory for density estimation . Wenliang et al . ( 2019 ) parametrized the kernel with a deep neural network . Other density estimation papers include Arbel & Gretton ( 2018 ) ; Dai et al . ( 2019 ) ; Sutherland et al . ( 2018 ) . We apply kernel exponential families as attention densities to weight a value function which represents the data , rather than to estimate the data density , and extend similar ideas to kernel deformed exponential families with sparse support . Wenliang et al . ( 2019 ) showed a condition for an unnormalized kernel exponential family density to have a finite normalizer . However , they used exponential power base densities . We instead relate kernel growth rates to the base density tail decay , allowing non-symmetric base densities . To summarize our theoretical contributions : 1 ) introducing kernel deformed exponential families with approximation and normalization analysis 2 ) improved kernel exponential family normalization results . 3 CONTINUOUS ATTENTION MECHANISMS . An attention mechanism involves : 1 ) the value function approximates a data representation . This may be the original data or a learned representation . 2 ) the attention density is chosen to be ’ similar ’ to another data representation , encoding it into a density 3 ) the context combines the two , taking an expectation of the value function with respect to the attention density . Formally , the context is c = ET∼p [ V ( T ) ] . ( 1 ) Here V ( t ) the value function approximates a data representation , T ∼ p ( t ) is the random variable or vector for locations ( temporal , spatial , etc ) , and p ( t ) is the attention density . To choose the attention density p , one takes a data representation f and finds p ’ similar ’ to f and thus to a data representation , but regularizing p. Martins et al . ( 2020 ; 2021 ) did this , providing a rigorous formulation of attention mechanisms . Given a probability space ( S , A , Q ) , letM1+ ( S ) be the set of densities with respect to Q . Assume that Q is dominated by a measure ν ( i.e . Lebesgue ) and that it has density q0 = dQdν with respect to ν . Let S ⊆ R D , F be a function class , and Ω : M1+ ( S ) → R be a lower semi-continuous , proper , strictly convex functional . Given f ∈ F , an attention density ( Martins et al. , 2020 ) p̂ : F → R≥0 solves p̂ [ f ] = arg max p∈M1+ ( S ) 〈p , f〉L2 ( Q ) − Ω ( p ) . ( 2 ) This maximizes regularized L2 similarity between p and a data representation f . If Ω ( p ) is the negative differential entropy , the attention density is Boltzmann Gibbs p̂ [ f ] ( t ) = exp ( f ( t ) −A ( f ) ) , ( 3 ) where A ( f ) ensures ∫ S p̂ [ f ] ( t ) dQ = 1 . If f ( t ) = θTφ ( t ) for parameters and statistics θ ∈ RM , φ ( t ) ∈ RM respectively , Eqn . 3 becomes an exponential family density . For f in a reproducing kernel Hilbert space H , it becomes a kernel exponential family density ( Canu & Smola , 2006 ) , which we propose to use as an alternative attention density . One desirable class would be heavy or thin tailed exponential family-like densities . In exponential families , the support , or non-negative region of the density , is controlled by the measure Q . Letting Ω ( p ) be the α-Tsallis negative entropy Ωα ( p ) ( Tsallis , 1988 ) , Ωα ( p ) = { 1 α ( α−1 ) ( ∫ S p ( t ) αdQ− 1 ) , α 6= 1 ; ∫ S p ( t ) log p ( t ) dQ , α = 1 , then p̂ [ f ] for f ( t ) = θTφ ( t ) lies in the deformed exponential family ( Tsallis , 1988 ; Naudts , 2004 ) p̂Ωα [ f ] ( t ) = exp2−α ( θ Tφ ( t ) −Aα ( f ) ) , ( 4 ) where Aα ( f ) again ensures normalization and the density uses the β-exponential expβ ( t ) = { [ 1 + ( 1− β ) t ] 1/ ( 1−β ) + , β 6= 1 ; exp ( t ) , β = 1 . ( 5 ) For β < 1 , Eqn . 5 and thus deformed exponential family densities for 1 < α ≤ 2 can return 0 values . Values α > 1 ( and thus β < 1 ) give thinner tails than the exponential family , while α < 1 gives fatter tails . Setting β = 0 is called sparsemax ( Martins & Astudillo , 2016 ) . In this paper , we assume 1 < α ≤ 2 , which is the sparse case studied in Martins et al . ( 2020 ) . We again propose to replace f ( t ) = θTφ ( t ) with f ∈ H , which leads to the novel kernel deformed exponential families . Computing Eqn . 1 ’ s context vector requires parametrizing V ( t ) . Martins et al . ( 2020 ) obtain a value function V : S → RD parametrized by B ∈ RD×N by applying regularized multivariate linear regression to estimate V ( t ; B ) = BΨ ( t ) , where Ψ = { ψn } Nn=1 is a set of basis functions . Let L be the number of observation locations ( times in a temporal setting ) , O be the observation dimension , andN be the number of basis functions . This involves regressing the observation matrix H ∈ RO×L on a matrix F ∈ RN×L of basis functions { ψn } Nn=1 evaluated at observation locations { tl } Ll=1 B∗ = arg min B ‖BF−H‖2F + λ‖B‖2F . ( 6 )

Summary: Many modern neural architectures, especially in natural language processing, rely heavily on the attention mechanism. Previous work in the literature proposed to extend the softmax-based attention mechanism by using different distribution families. In particular, the authors of this paper focus on variants of the attention mechanism that allow for continuous and sparse attention.  The authors propose kernel deformed exponential families, an extension of the exponential family that allows sparse and multimodal attention, contrary to most of previous work that focused on unimodal attention.