Year: 2021

Abstract:
Attention mechanisms take an expectation of a data representation with respect to probability weights. This creates summary statistics that focus on important features. Recently, Martins et al. (2020; 2021) proposed continuous attention mechanisms, focusing on unimodal attention densities from the exponential and deformed exponential families: the latter has sparse support. Farinhas et al. (2021) extended this to use Gaussian mixture attention densities, which are a flexible class with dense support. In this paper, we extend this to two general flexible classes: kernel exponential families (Canu & Smola, 2006) and our new sparse counterpart kernel deformed exponential families. Theoretically, we show new existence results for both kernel exponential and deformed exponential families, and that the deformed case has similar approximation capabilities to kernel exponential families. Experiments show that kernel deformed exponential families can attend to multiple compact regions of the data domain.

heading: 1 INTRODUCTION
text: Attention mechanisms take weighted averages of data representations (Bahdanau et al., 2015), where the weights are a function of input objects. These are then used as inputs for prediction. Discrete attention 1) cannot easily handle data where observations are irregularly spaced 2) attention maps may be scattered, lacking focus. Martins et al. (2020; 2021) extended attention to continuous settings, showing that attention densities maximize the regularized expectation of a function of the data location (i.e. time). Special case solutions lead to exponential and deformed exponential families: the latter has sparse support. They form a continuous data representation and take expectations with respect to attention densities. Using measure theory to unify discrete and continuous approaches, they show transformer self-attention (Vaswani et al., 2017) is a special case of their formulation.
Martins et al. (2020; 2021) explored unimodal attention densities: these only give high importance to one region of data. Farinhas et al. (2021) extended this to use multi-modal mixture of Gaussian attention densities. However 1) mixture of Gaussians do not lie in either the exponential or deformed exponential families, and are difficult to study in the context of Martins et al. (2020; 2021) 2) they have dense support. Sparse support can say that certain regions of data do not matter: a region of time has no effect on class probabilities, or a region of an image is not some object. We would like to use multimodal exponential and deformed exponential family attention densities, and understand how Farinhas et al. (2021) relates to the optimization problem of Martins et al. (2020; 2021).
This paper makes three contributions: 1) we introduce kernel deformed exponential families, a multimodal class of densities with sparse support and apply it along with the multimodal kernel exponential families (Canu & Smola, 2006) to attention mechanisms. The latter have been used for density estimation, but not weighting data importance 2) we theoretically analyze normalization for both kernel exponential and deformed exponential families in terms of a base density and kernel, and show approximation properties for the latter 3) we apply them to real world datasets and show that kernel deformed exponential families learn flexible continuous attention densities with sparse support. Approximation properties for the kernel deformed are challenging: similar kernel exponential family results (Sriperumbudur et al., 2017) relied on standard exponential and logarithm properties to bound the difference of the log-partition functional at two functions: these do not hold for deformed analogues. We provide similar bounds via the functional mean value theorem along with bounding the Frechet derivative of the deformed log-partition functional.
The paper is organized as follows: we review continuous attention (Martins et al., 2020; 2021). We then describe how mixture of Gaussian attention densities, used in Farinhas et al. (2021), solve a different optimization problem. We next describe kernel exponential families and give novel normalization condition relating the kernel growth to the base density’s tail decay. We then propose kernel deformed exponential families, which can have support over disjoint regions. We describe normalization and prove approximation capabilities. Next we describe use of these densities for continuous attention, including experiments where we show that the kernel deformed case learns multimodal attention densities with sparse support. We conclude with limitations and future work.

heading: 2 RELATED WORK
text: Attention Mechanisms closely related are Martins et al. (2020; 2021); Farinhas et al. (2021); Tsai et al. (2019); Shukla & Marlin (2021). Martins et al. (2020; 2021) frame continuous attention as an expectation of a value function over the domain with respect to a density, where the density solves an optimization problem. They only used unimodal exponential and deformed exponential family densities: we extend this to the multimodal setting by leveraging kernel exponential families and proposing a deformed counterpart. Farinhas et al. (2021) proposed a multi-modal continuous attention mechanism via a mixture of Gaussians approach. We show that this solves a slightly different optimization problem from Martins et al. (2020; 2021), and extend to two further general density classes. Shukla & Marlin (2021) provide an attention mechanism for irregularly sampled time series by use of a continuous-time kernel regression framework, but do not actually take an expectation of a data representation over time with respect to a continuous pdf, evaluating the kernel regression model at a fixed set of time points to obtain a discrete representation. This describes importance of data at a set of points rather than over regions. Other papers connect attention and kernels, but focus on the discrete attention setting (Tsai et al., 2019; Choromanski et al., 2020). Also relevant are temporal transformer papers, including Xu et al. (2019); Li et al. (2019; 2020); Song et al. (2018). However none have continuous attention densities.
Kernel Exponential Families Canu & Smola (2006) proposed kernel exponential families: Sriperumbudur et al. (2017) analyzed theory for density estimation. Wenliang et al. (2019) parametrized the kernel with a deep neural network. Other density estimation papers include Arbel & Gretton (2018); Dai et al. (2019); Sutherland et al. (2018). We apply kernel exponential families as attention densities to weight a value function which represents the data, rather than to estimate the data density, and extend similar ideas to kernel deformed exponential families with sparse support.
Wenliang et al. (2019) showed a condition for an unnormalized kernel exponential family density to have a finite normalizer. However, they used exponential power base densities. We instead relate kernel growth rates to the base density tail decay, allowing non-symmetric base densities.
To summarize our theoretical contributions: 1) introducing kernel deformed exponential families with approximation and normalization analysis 2) improved kernel exponential family normalization results.

heading: 3 CONTINUOUS ATTENTION MECHANISMS
text: An attention mechanism involves: 1) the value function approximates a data representation. This may be the original data or a learned representation. 2) the attention density is chosen to be ’similar’ to another data representation, encoding it into a density 3) the context combines the two, taking an expectation of the value function with respect to the attention density. Formally, the context is
c = ET∼p[V (T )]. (1) Here V (t) the value function approximates a data representation, T ∼ p(t) is the random variable or vector for locations (temporal, spatial, etc), and p(t) is the attention density.
To choose the attention density p, one takes a data representation f and finds p ’similar’ to f and thus to a data representation, but regularizing p. Martins et al. (2020; 2021) did this, providing a rigorous formulation of attention mechanisms. Given a probability space (S,A, Q), letM1+(S) be the set of densities with respect to Q. Assume that Q is dominated by a measure ν (i.e. Lebesgue) and that it has density q0 = dQdν with respect to ν. Let S ⊆ R
D, F be a function class, and Ω :M1+(S) → R be a lower semi-continuous, proper, strictly convex functional.
Given f ∈ F , an attention density (Martins et al., 2020) p̂ : F → R≥0 solves
p̂[f ] = arg max p∈M1+(S)
〈p, f〉L2(Q) − Ω(p). (2)
This maximizes regularized L2 similarity between p and a data representation f . If Ω(p) is the negative differential entropy, the attention density is Boltzmann Gibbs
p̂[f ](t) = exp(f(t)−A(f)), (3) where A(f) ensures ∫ S p̂[f ](t)dQ = 1. If f(t) = θTφ(t) for parameters and statistics θ ∈ RM , φ(t) ∈ RM respectively, Eqn. 3 becomes an exponential family density. For f in a reproducing kernel Hilbert space H, it becomes a kernel exponential family density (Canu & Smola, 2006), which we propose to use as an alternative attention density.
One desirable class would be heavy or thin tailed exponential family-like densities. In exponential families, the support, or non-negative region of the density, is controlled by the measure Q. Letting Ω(p) be the α-Tsallis negative entropy Ωα(p) (Tsallis, 1988),
Ωα(p) =
{ 1 α(α−1) (∫ S p(t)αdQ− 1 ) , α 6= 1;∫
S p(t) log p(t)dQ, α = 1,
then p̂[f ] for f(t) = θTφ(t) lies in the deformed exponential family (Tsallis, 1988; Naudts, 2004)
p̂Ωα [f ](t) = exp2−α(θ Tφ(t)−Aα(f)), (4)
where Aα(f) again ensures normalization and the density uses the β-exponential
expβ(t) = { [1 + (1− β)t]1/(1−β)+ , β 6= 1; exp(t), β = 1.
(5)
For β < 1, Eqn. 5 and thus deformed exponential family densities for 1 < α ≤ 2 can return 0 values. Values α > 1 (and thus β < 1) give thinner tails than the exponential family, while α < 1 gives fatter tails. Setting β = 0 is called sparsemax (Martins & Astudillo, 2016). In this paper, we assume 1 < α ≤ 2, which is the sparse case studied in Martins et al. (2020). We again propose to replace f(t) = θTφ(t) with f ∈ H, which leads to the novel kernel deformed exponential families. Computing Eqn. 1’s context vector requires parametrizing V (t). Martins et al. (2020) obtain a value function V : S → RD parametrized by B ∈ RD×N by applying regularized multivariate linear regression to estimate V (t; B) = BΨ(t), where Ψ = {ψn}Nn=1 is a set of basis functions. Let L be the number of observation locations (times in a temporal setting), O be the observation dimension, andN be the number of basis functions. This involves regressing the observation matrix H ∈ RO×L on a matrix F ∈ RN×L of basis functions {ψn}Nn=1 evaluated at observation locations {tl}Ll=1
B∗ = arg min B ‖BF−H‖2F + λ‖B‖2F . (6)

heading: 3.1 GAUSSIAN MIXTURE MODEL
text: Farinhas et al. (2021) used mixture of Gaussian attention densities, but did not relate this to the optimization definition of attention densities in Martins et al. (2020; 2021). In fact their attention densities solve a related but different optimization problem. Martins et al. (2020; 2021) show that exponential family attention densities maximize a regularized linear predictor of the expected sufficient statistics of locations. In contrast, Farinhas et al. (2021) find a joint density over locations and latent states, and maximize a regularized linear predictor of the expected joint sufficient statistics. They then take the marginal location densities to be the attention densities.
Let Ω(p) be Shannon entropy and consider two optimization problems:
arg max p∈M1+(S)
〈θ,Ep[φ(T )]〉l2 − Ω(p)
arg max p∈M1+(S)
〈θ,Ep[φ(T,Z)]〉l2 − Ω(p)
The first is Eqn. 2 with f = θTφ(t) and rewritten to emphasize expected sufficient statistics. If one solves the second with variables Z, we recover an Exponential family joint density
p̂Ωα [f ](t, z) = exp(θ Tφ(t, z)−A(θ)).
This encourages the joint density of T,Z to be similar to a complete data representation θTφ(t, z) of both location variables T and latent variables Z, instead of encouraging the density of T to be similar to an observed data representation θTφ(t). The latter optimization is equivalent to
arg max p∈M1+(S) Ω(p)
s.t. Ep(T,Z)[φm(T,Z)] = cm,m = 1, · · · ,M.
The constraint terms cm are determined by θ. Thus, this maximizes the joint entropy of Z and T , subject to constraints on the expected joint sufficient statistics.
To recover EM learned Gaussian mixture densities, one must select φm so that the marginal distribution of T will be a mixture of Gaussians, and relate cm to the EM algorithm used to learn the mixture model parameters. For the first, assume that Z is a multinomial random variable taking |Z| possible values and let φ(t, z) = (z1, z2, · · · , z|Z|−1, I(z = 1)t, I(z = 1)t2, · · · , I(z = |Z|)t, I(z = |Z|)t2). These are multinomial sufficient statistics, followed by the sufficient statistics of |Z| Gaussians multiplied by indicators for each z. Then p(T |Z) will be Gaussian, p(Z) will be multinomial, and p(T ) will be a Gaussian mixture. For contraints, Farinhas et al. (2021) have
Ep(T,Z)[φm(T,Z)] = L∑ l=1 wl |Z|∑ z=1 pold(z|tl)φm(tl, z),m = 1, · · · ,M (7)
at each EM iteration. Here pold(z|xl) is the previous iteration’s latent state density conditional on the observation value, wl are discrete attention weights, and tl is a discrete attention location. That EM has this constraint was shown in Wang et al. (2012). Intuitively, this matches the expected joint sufficient statistics to those implied by discrete attention over locations, taking into account the dependence between z and tl given by old model parameters. An alternative is simply to let θ be the output of a neural network. While the constraints lack the intuition of Eqn. 7, it avoids the need to select an initialization. We focus on this case and use it for our baselines: both approaches are valid.

heading: 4 KERNEL EXPONENTIAL AND DEFORMED EXPONENTIAL FAMILIES
text: We now use kernel exponential families and a new deformed counterpart to obtain flexible attention densities solving Eqn. 2 with the same regularizers. We first review kernel exponential families. We then give a novel theoretical result describing when an unnormalized kernel exponential family density can be normalized. Next we introduce kernel deformed exponential families, extending kernel exponential families to have either sparse support or fatter tails: we focus on the former. These can attend to multiple non-overlapping time intervals or spatial regions. We show similar normalization results based on the choice of kernel and base density. Following this we show approximation theory. We conclude by showing how to compute attention densities in practice.
Kernel exponential families (Canu & Smola, 2006) extend exponential family distributions, replacing f(t) = θTφ(t) with f in a reproducing kernel Hilbert space H (Aronszajn, 1950) with kernel k : S × S → R. Densities can be written as
p(t) = exp(f(t)−A(f)) = exp(〈f, k(·, t)〉H〉 −A(f)),
where the second equality follows from the reproducing property. A challenge is to choose H, Q so that a normalizing constant exists, i.e., ∫ S
exp(f(t))dQ < ∞. Kernel exponential family distributions can approximate any continuous density over a compact domain arbitrarily well in KL divergence, Hellinger, and Lp distance (Sriperumbudur et al., 2017). However relevant integrals including the normalizing constant generally require numerical integration.
To avoid infinite dimensionality one generally assumes a representation of the form
f = I∑ i=1 γik(·, ti),
where for density estimation (Sriperumbudur et al., 2017) the ti are the observation locations. However, this requires using one parameter per observation value. This level of model complexity may not be necessary, and often one chooses a set of inducing points (Titsias, 2009) {ti}Ii=1 where I is less than the number of observation locations.
For a given pair H, k, how can we choose Q to ensure that the normalization constant exists? We first give a simple example ofH, f and Q where the normalizing constant does not exist. Example 1. Let Q be the law of a N (0, 1) distribution and S = R. Let H = span{t3, t4} with k(x, y) = x3y3 + x4y4 and f(t) = t3 + t4 = k(t, 1). Then∫
S
exp(f(t))dQ = ∫ R exp( t2 2 + t3 + t4)dt (8)
where the integral diverges.

heading: 4.1 THEORY FOR KERNEL EXPONENTIAL FAMILIES
text: We provide sufficient conditions for Q and H so that A(f) the log-partition function exists. We relateH’s kernel growth rate to the tail decay of the random variable or vector TQ with law Q. Proposition 1. Let p̃(t) = exp(f(t)) where f ∈ H an RKHS with kernel k. Assume k(t, t) ≤ Lk‖t‖ξ2 + Ck for constants Lk, Ck, ξ > 0. Let Q be the law of a random vector TQ, so that Q(A) = P (TQ ∈ A). Assume ∀u s.t. ‖u‖2 = 1,
P (|uTTQ| ≥ z) ≤ Cq exp(−vzη) (9)
for some constants η > ξ2 , CQ, v > 0. Then∫ S p̃(t)dQ <∞.
Proof. See Appendix A.1
Based on k(t, t)’s growth, we can vary what tail decay rate for TQ ensures we can normalize p̃(t). Wenliang et al. (2019) also proved normalization conditions, but focused on random variables with exponential power density for a specific growth rate of k(t, t) rather than relating tail decay to growth rate. By focusing on tail decay, our result can be applied to non-symmetric base densities. Specific kernel bound growth rate terms ξ lead to allowing different tail decay rates. Corollary 1. For ξ = 4, TQ can be any sub-Gaussian random vector. For ξ = 2 it can be any sub-exponential. For ξ = 0 it can have any density.
Proof. See Appendix A.2

heading: 4.2 KERNEL DEFORMED EXPONENTIAL FAMILIES
text: We now propose kernel deformed exponential families: flexible sparse non-parametric distributions. These take deformed exponential families and extend them to use kernels in the deformed exponential term. This mirrors kernel exponential families. We write
p(t) = exp2−α(f(t)−Aα(f)),
where f ∈ H with kernel k. Fig. 1b shows that they can have support over disjoint intervals.

heading: 4.2.1 NORMALIZATION THEORY
text: We construct a valid kernel deformed exponential family density from Q and f ∈ H. We first discuss the deformed log normalizer. In exponential family densities, the log-normalizer is the log of the normalizer. For deformed exponentials, the following holds.
Lemma 1. Let Z > 0 be a constant. Then for 1 < α ≤ 2, 1
Z exp2−α(Z
α−1f(t)) = exp2−α(f(t)− logα Z)
where
logβ t =  t1−β−1 1−β if t > 0, β 6= 1; log(t) if t > 0, β = 1; undefined if t ≤ 0.
Proof. See Appendix B.1
We describe a normalization sufficient condition analagous to Proposition 1 for the sparse deformed kernel exponential family. With Lemma 1, we can take an unnormalized exp2−α(f̃(t)) and derive a valid normalized kernel deformed exponential family density. We only require that an affine function of the terms in the deformed-exponential are negative for large magnitude t.
Proposition 2. For 1 < α ≤ 2 assume p̃(t) = exp2−α(f̃(t)) with f̃ ∈ H,H is a RKHS with kernel k. If ∃Ct > 0 s.t. for ‖t‖2 > Ct, (α− 1)f̃(t) + 1 ≤ 0 and k(t, t) ≤ Lk‖t‖ξ2 + Ck for some ξ > 0, then ∫ S exp2−α(f̃(t))dQ <∞.
Proof. See Appendix B.2
We now construct a valid kernel deformed exponential family density using the finite integral.
Corollary 2. Under the conditions of proposition 2, assume exp2−α(f̃(t)) > 0 on a setA ⊆ S such that Q(A) > 0, then ∃ constants Z > 0, Aα(f) ∈ R such that for f(t) = 1Zα−1 f̃(t), the following holds ∫
S
exp2−α(f(t)−Aα(f))dQ = 1.
Proof. See Appendix B.3.
We thus estimate f̃(t) = (Z)α−1f(t) and normalize to obtain a density of the desired form.

heading: 4.2.2 APPROXIMATION THEORY
text: Under certain kernel conditions, kernel deformed exponential family densities can approximate densities of a similar form where the RKHS function is replaced with a C0(S) 1 function.
1continuous function on domain S vanishing at infinity
Proposition 3. Define
P0 = {πf (t) = exp2−α(f(t)−Aα(f)), t ∈ S : f ∈ C0(S)}
where S ⊆ Rd. Suppose k(x, ·) ∈ C0(S),∀x ∈ S and∫ ∫ k(x, y)dµ(x)dµ(y) > 0,∀µ ∈Mb(S)\{0}. (10)
hereMb(S) is the space of bounded measures over S. Then the set of deformed exponential families is dense in P0 wrt Lr(Q) norm and Hellinger distance.
Proof. See Appendix B.4
We apply this to approximate fairly general densities with kernel deformed exponential families.
Theorem 1. Let q0 ∈ C(S), such that q0(t) > 0 for all t ∈ S, where S ⊆ Rd is locally compact Hausdorff and q0(t) is the density ofQ with respect to a dominating measure ν. Suppose there exists l > 0 such that for any > 0,∃R > 0 satisfying |p(t)− l| ≤ for any t with ‖t‖2 > R. Define
Pc = {p ∈ C(S) : ∫ S p(t)dQ = 1, p(t) ≥ 0,∀t ∈ S and p− l ∈ C0(S)}.
Suppose k(t, ·) ∈ C0(S)∀t ∈ S and the kernel integration condition (Eqn. 10) holds. Then kernel deformed exponential families are dense in Pc wrt Lr norm, Hellinger distance and Bregman divergence for the α-Tsallis negative entropy functional.
Proof. See Appendix B.5.
For uniform q0, kernel deformed exponential families can thus approximate continuous densities on compact domains arbitrarily well. Our Bregman divergence result is analagous to the KL divergence result in Sriperumbudur et al. (2017). KL divergence is Bregman divergence with the Shannon entropy functional: we show the same for Tsallis entropy. The Bregman divergence here describes how close the uncertainty in a density is to its first order approximation evaluated at another density. Using the Tsallis entropy functional here is appropriate for deformed exponential families: they maximize it given expected sufficient statistics (Naudts, 2004).
These results extend Sriperumbudur et al. (2017)’s approximation results to the deformed setting, where standard log and exponential rules cannot be applied. The Bregman divergence case requires bounding Frechet derivatives and applying the functional mean value theorem.

heading: 4.3 USING KERNELS FOR CONTINUOUS ATTENTION
text: We apply kernel exponential and deformed exponential families to attention. The forward pass computes attention densities and the context vector. The backwards pass uses automatic differentiation. We assume a vector representation v ∈ R|v| computed from the locations we take an expectation over. For kernel exponential families we compute kernel weights {γi}Ii=1 for f(t) = ∑I i=1 γik(t, ti)
γi = w T i v, and compute Z = ∫ S
exp(f(t))dQ numerically. For the deformed case we compute γ̃i = wTi v and f̃(t) = (Z)α−1f(t) = ∑I i=1 γ̃ik(t, ti) followed by Z = ∫ S exp2−α(f̃(t))dQ. The context
c = ET∼p[V (T )] = BEp[Ψ(t)]
requires taking the expectation of Ψ(T ) with respect to a (possibly deformed) kernel exponential family density p. Unlike Martins et al. (2020; 2021), where they obtained closed form expectations, difficult normalizing constants prevent us from doing so. We thus use numerical integration for the forward pass and automatic differentiation for the backward pass. Algorithm 1 shows how to compute a continuous attention mechanism for a kernel deformed exponential family attention density. The kernel exponential family case is similar.
Algorithm 1 Continuous Attention Mechanism via Kernel Deformed Exponential Families Choose base density q0(t) and kernel k. Inducing point locations {ti}Ii=1 Input Vector representation v of input object i.e. document representation Parameters {γ̃i}Ii=1 the weights for f̃(t) = (Z)α−1f(t) = ∑I i=1 γ̃ik(t, ti), matrix B for basis

heading: 5 EXPERIMENTS
text: For document classification, we follow Martins et al. (2020)’s architecture. For the remaining, architectures have four parts: 1) an encoder takes a discrete representation of a time series and outputs attention density parameters. 2) The value function takes a time series representation (original or after passing through a neural network) and does (potentially multivariate) linear regression to obtain parameters B for a function V (t; B). These are combined to compute 3) context c = Ep[V (T )], which is used in a 4) classifier. Fig. 2 in the Appendices visualizes this.

heading: 5.1 DOCUMENT CLASSIFICATION
text: We extend Martins et al. (2020)’s code2 for the IMDB sentiment classification dataset (Maas et al., 2011). This starts with a document representation v computed from a convolutional neural network and uses an LSTM attention model. We use a Gaussian base density and kernel, and divide the interval [0, 1] into I = 10 inducing points where we evaluate the kernel in f(t) = ∑I i=1 γik(t, ti). We set the bandwidth to be 0.01 for I = 10. Table 1 shows results. On average, kernel exponential and deformed exponential family slightly outperforms the continuous softmax and sparsemax, although the results are essentially the same. The continuous softmax/sparsemax results are from their code.
2Martins et al. (2020)’s repository for this dataset is https://github.com/deep-spin/quati

heading: 5.2 UWAVE DATASET
text: We analyze uWave (Liu et al., 2009): accelerometer time series with eight gesture classes. We follow Li & Marlin (2016)’s split into 3,582 training observations and 896 test observations: sequences have length 945. We do synthetic irregular sampling and uniformly sample 10% of the observations. Table 2 shows results. Our highest accuracy is 94.26%, the unimodal case’s best is 74.69%, and the mixture’s best is 81.13%. Since this dataset is small, we report ±1.96 standard deviations from 10 runs. Fig. 1 shows that attention densities have support over non-overlapping time intervals. This cannot be done with Gaussian mixtures, and the intervals would be the same for each density in the exponential family case. Appendix C describes additional details.

heading: 6 ECG HEARTBEAT CLASSIFICATION
text: We use the MIT Arrhythmia Database’s (Goldberger et al., 2000) Kaggle 3. The task is to detect abnormal heart beats from ECG signals. The five different classes are {Normal, Supraventricular premature, Premature ventricular contraction, Fusion of ventricular and normal, Unclassifiable}. There are 87,553 training samples and 21,891 test samples. Our value function is trained using the output of repeated convolutional layers: the final layer has 256 dimensions and 23 time points. Our encoder is a feedforward neural network with the original data as input, and our classifier is a feedforward network. Table 3 shows results. All accuracies are very similar, but the F1 score of kernel sparsemax is drastically higher. Additional details are in Appendix D.

heading: 7 DISCUSSION
text: In this paper we extend continuous attention mechanisms to use kernel exponential and deformed exponential family attention densities. The latter is a new flexible class of non-parametric densities with sparse support. We show novel existence properties for both kernel exponential and deformed exponential families, and prove approximation properties for the latter. We then apply these to the framework described in Martins et al. (2020; 2021) for continuous attention. We show results on three datasets: sentiment classification, gesture recognition, and arrhythmia classification. In the first case performance is similar to unimodal attention, for the second it is drastically better, and in the third it is similar in the dense case and drastically better in the sparse case.

heading: 7.1 LIMITATIONS AND FUTURE WORK
text: A limitation of this work was the use of numerical integration, which scales poorly with the dimensionality of the locations. Because of this we restricted our applications to temporal and text data. This still allows for multiple observation dimensions at a given location. A future direction would be to use varianced reduced Monte Carlo to approximate the integral, as well as studying how to choose the number of basis functions in the value function and how to choose the number of inducing points.
3https://www.kaggle.com/mondejar/mitbih-database

heading: A PROOF RELATED TO PROPOSITION 1

heading: A.1 PROOF OF PROPOSITION 1
text: Proof. This proof has several parts. We first bound the RKHS function f and use the general tail bound we assumed to give a tail bound for the one dimensional marginals TQd of TQ. Using the RKHS function bound, we then bound the integral of the unnormalized density in terms of expectations with respect to these finite dimensional marginals. We then express these expectations over finite dimensional marginals as infinite series of integrals. For each integral within the infinite series, we use the finite dimensional marginal tail bound to bound it, and then use the ratio test to show that the infinite series converges. This gives us that the original unnormalized density has a finite integral.
We first note, following Wenliang et al. (2019), that the bound on the kernel in the assumption allows us to bound f in terms of two constants and the absolute value of the point at which it is evaluated.
|f(t)| = |〈f, k(t, ·)〉H| reproducing property ≤ ‖f‖H‖k(t, ·)‖H Cauchy Schwarz
= ‖f‖H √ 〈k(t, ·), k(t, ·)〉H
= ‖f‖H √ k(t, t)
≤ ‖f‖H √ Lk‖t‖ξ + Ck by assumption
≤ C0 + C1‖t‖|ξ|/2 for some C1, C2 > 0.
We can write TQ = (TQ1, · · · , TQD). Let ed be a standard Euclidean basis vector. Then by the assumption and setting u = ed we have
P (|TQd| ≥ z) ≤ Cq exp(−vzη)
Letting Qd be the marginal law,∫ S exp(f(t))dQ ≤ ∫ S exp(C0 + C1‖t‖ξ/2)dQ
= exp(C0) ∫ S exp(C1‖t‖ξ/2)dQ
= exp(C0)E exp(C1‖TQ‖ξ/2) ≤ exp(C0)E exp(C1( √ d max d=1,··· ,D |TQd|)ξ/2) ≤ exp(C0) D∑ d=1 E exp(C2|TQd|ξ/2)
which will be finite if each E exp(C2|TQd|ξ/2) < ∞. Now letting Sd be the relevant dimension of S,
E exp(C2|TQd|ξ/2) = ∫ Sd exp(C2|td|ξ/2)dQd
≤ −1∑
j=−∞
∫ j+1 j exp(C2|td|ξ/2)dQd + ∞∑ j=0 ∫ j+1 j exp(C2|td|ξ/2)dQd
where the inequality follows since Sd ⊆ R, exp is a non-negative function and probability measures are monotonic. We will show that the second sum converges. Similar techniques can be shown for the first sum. Note that for j ≥ 0
Qd([j, j + 1)) = P (Td ≥ j)− P (Td ≥ j + 1) ≤ P (Td ≥ j) ≤ Cq exp(−vjη) by assumption
Then ∞∑ j=0 ∫ j+1 j exp(C2|td|ξ/2)dQd ≤ ∞∑ j=0 exp(C2|j|ξ/2)Qd([j, j + 1))
≤ ∞∑ j=0 CQ exp(C2|j|ξ/2 − vjη)
Let aj = exp(C2|j|ξ/2 − viη). We will use the ratio test to show that the RHS converges. We have∣∣∣∣aj+1aj ∣∣∣∣ = exp(C2((j + 1)ξ/2 − jξ/2)− v[(j + 1)η − jη]). (11)
We want this ratio to be < 1 for large j. We thus need to select η so that for sufficiently large j, we have
C1 v ((j + 1)ξ/2 − jξ/2) < [(j + 1)η − jη].
Assume that η > ξ2 . Then
(j + 1)η − jη
(j + 1)ξ/2 − jξ/2 =
jη[(1 + 1j ) η − 1]
jξ/2[(1 + 1j ) ξ/2 − 1]
≥ jη−ξ/2.
Since the RHS is unbounded for η > ξ2 , we have that Eqn. 11 holds for sufficiently large j. By the ratio test Eqd(t) exp(C2|Td|ξ/2) = ∑−1 j=−∞ ∫ j+1 j
exp(C2|td|ξ/2)dQd +∑∞ j=0 ∫ j+1 j
exp(C2|td|ξ/2)dQd is finite. Thus putting everything together we have∫ S exp(f(t))dQ ≤ ∫ S exp(C0 + C1‖t‖ξ/2)dQ
< exp(C0) D∑ d=1 E exp(C2|TQd|ξ/2)
<∞
and p̃(t) can be normalized.

heading: A.2 PROOF OF COROLLARY 1
text: Proof. Let ξ = 4. Then η > 2 and
P (|uTT | > t) ≤ P (|uTT | ≥ t) monotonicity ≤ CQ exp(−vtη) < CQ exp(−vt2).
The second case is similar. For the uniformly bounded kernel,∫ S exp(〈f, k(·, t)〉H)dQ ≤ exp(‖f‖H √ Ck) ∫ S dQ
= exp(‖f‖H √ Ck)
<∞.
The first line follows from Cauchy Schwarz and ξ = 0

heading: B PROOFS RELATED TO KERNEL DEFORMED EXPONENTIAL FAMILY

heading: B.1 PROOF OF LEMMA 1
text: Proof. The high level idea is to express a term inside the deformed exponential family that becomes 1/Z once outside.
exp2−α(f(t)− logα(Z)) = [1 + (α− 1)(f(t)− logα Z)] 1 α−1 +
= [1 + (α− 1)f(t)− (α− 1)Z 1−α − 1 1− α ] 1 α−1 +
= [1 + (α− 1)f(t) + Z1−α − 1] 1
α−1 +
= [(α− 1)f(t) + Z1−α] 1
α−1 +
= [(α− 1)f(t)Z α−1
Zα−1 + Z1−α)]
1 α−1 +
= 1 Z [(α− 1)f(t) 1 Zα−1 + 1] 1 α−1 + = 1
Z exp2−α(Z
α−1f(t))

heading: B.2 PROOF OF PROPOSITION 2

heading: Proof. ∫
text: S exp2−α(f̃(t))dQ = ∫ S [1 + (α− 1)f̃(t)] 1 α−1 + dQ
= ∫ ‖t‖≤Ct [1 + (α− 1)f̃(t)] 1 α−1 + dQ
≤ ∫ ‖t‖≤Ct [1 + (α− 1)(C0 + C1|Ct|ξ/2)] 1 α−1 + dQ
<∞

heading: B.3 PROOF OF COROLLARY 2
text: Proof. From proposition 2 and the assumption,∫ S exp2−α(f̃(t))dQ = Z
for some Z > 0. Then ∫ S 1 Z exp2−α(Z
α−1f(t))dQ = 1∫ S exp2−α(f(t)− logα Z)dQ = 1
where the second line follows from lemma 1. Set Aα(f) = logα(Z) and we are done.

heading: B.4 PROOF OF PROPOSITION 3
text: Proof. The kernel integration condition tells us that H is dense in C0(S) with respect to L∞ norm. This was shown in Sriperumbudur et al. (2011). For the Lr norm, we apply ‖pf − pg‖Lr ≤ 2Mexp‖f − g‖∞ from Lemma 5 with f ∈ C0(S), g ∈ H, and f0 = f . L1 convergence implies Hellinger convergence.

heading: B.5 PROOF OF THEOREM 1
text: Proof. For any p ∈ Pc, define pδ = p+δ1+δ . Then
‖p− pδ‖r = δ
1 + δ ‖p‖r
→ 0
for 1 ≤ r ≤ ∞. Thus for any > 0,∃δ > 0 such that for any 0 < θ < δ , we have ‖p− pθ‖r ≤ , where pθ(t) > 0 for all t ∈ S.
Define f = (
1+θ l+θ )1−α log2−α pθ 1+θ l+θ . Since p ∈ C(S), so is f . Fix any η > 0 and note that
f(t) ≥ η( 1 + θ
l + θ
)1−α log2−α pθ 1 + θ
l + θ ≥ η
log2−α pθ 1 + θ l + θ ≥ ( 1 + θ l + θ )α−1 η
pθ 1 + θ
l + θ ≥ exp2−α
(( 1 + θ
l + θ
)α−1 η )
pθ ≥ l + θ
1 + θ exp2−α
(( 1 + θ
l + θ
)α−1 η )
p− l ≥ (l + θ) ( exp2−α (( 1 + θ
l + θ
)α−1 η ) − 1 ) Thus
A = {t : f(t) ≥ η}
= { p− l ≥ (l + θ) ( exp2−α (( 1 + θ
l + θ
)α−1 η ) − 1 )}
Since p − l ∈ C0(S) the set on the second line is bounded. Thus A is bounded so that f ∈ C0(S). Further, by Lemma 1
pθ = exp2−α ( f − logα 1 + θ
l + θ ) giving us pθ ∈ P0. By Proposition 3 there is some pg in the deformed kernel exponential family so that ‖pθ − pg‖Lr(S) ≤ . Thus ‖p − pg‖r ≤ 2 for any 1 ≤ r ≤ ∞. To show convergence in Helinger distance, note
H2(p, pg) = 1
2 ∫ S ( √ p−√pg)2dQ
= 1
2 ∫ S (p− 2√ppg + pg)dQ
≤ 1 2 ∫ S (p− 2 min(p, pg) + pg)dQ
= 1
2 ∫ S |p− pg|dQ
= 1
2 ‖p− pg‖1
4actually an equality, see https://www2.cs.uic.edu/ zhangx/teaching/bregman.pdf for proof
so that L1(S) convergence, which we showed, implies Hellinger convergence. Let us consider the Bregman divergence. Note the generalized triangle inequality4 for Bregman divergence
BΩα(p, pg) = BΩα(p, pθ)︸ ︷︷ ︸ I +BΩα(pθ, pg)︸ ︷︷ ︸ II −〈p− pθ,∇Ωα(pθ)−∇Ωα(pg)〉2︸ ︷︷ ︸ III
(12)
Term I
BΩα(p, pθ) = 1
α(α− 1) ∫ S (pα − pαθ )dQ− 〈∇Ωα(pθ), p− pθ〉
= 1
α(α− 1) ∫ S (pα − pαθ )dQ− 1 α− 1 ∫ pα−1θ (p− pθ)dQ
≤ 1 α(α− 1) ∫ S (pα − pαθ )dQ+ 1 α− 1 ‖pα−1θ ‖1‖‖p− pθ‖∞
The first term on the rhs clearly vanishes as θ → 0. For the second term, we already showed that ‖p− pθ‖∞ → 0. Term II
Fix θ. Then term II converges to 0 by Lemma 5.
Term III
For term III , 〈p− pθ,∇Ωα(pθ)−∇Ωα(pg)〉2 ≤ ‖p− pθ‖∞‖∇Ωα(pθ)−∇Ωα(pg)‖1
Clearly the first term on the rhs converges by Lr convergence. The L1 term for the gradient is given by
‖∇Ωα(pθ)−∇Ωα(pg)‖1 = 1
α− 1
∫ |pθ(t)α−1 − pg(t)α−1|dQ
≤ ∫ (‖pθ‖∞ + ‖pθ − pg‖∞)α−2‖pθ − pg‖∞dQ Eqn. 17
= (‖pθ‖∞ + ‖pθ − pg‖∞)α−2‖pθ − pg‖∞ so that the inner product terms are bounded as |〈p− pθ,∇Ωα(pθ)−∇Ωα(pg)〉2| ≤ (‖pθ‖∞ + ‖pθ − pg‖∞)α−2‖pθ − pg‖∞‖p− pθ‖∞
Lemma 2. (Functional Taylor’s Theorem) Let F : X → R where X is a Banach space. Let f, g ∈ X and let F be k times Gateaux differentiable. Then we can write
F (g) = k−1∑ i=0 F i(f)(g − f)i + F k(f + c(g − f))(g − f)k
for some c ∈ [0, 1].
Proof. This is simply a consequence of expressing a functional as a function of an ∈ [0, 1], which restricts the functional to take input functions between two functions. We then apply Taylor’s theorem to the function and apply the chain rule for Gateaux derivatives to obtain the resulting Taylor remainder theorem for functionals.
Let G(η) = F (f + η(g − f)). By Taylor’s theorem we have G(1) = G(0) +G′(0) + · · ·+Gk(c)
and applying the chain rule gives us
F (g) = k−1∑ i=0 F i(f)(g − f)i + F k(f + c(g − f))(g − f)k
Lemma 3. (Functional Mean Value Theorem) Let F : X → R be a functional where f, g ∈ X some Banach space with norm ‖ · ‖. Then
|F (f)− F (g)| ≤ ‖F ′(h)‖op‖f − g‖
where h = g + c(f − g) for some c ∈ [0, 1], F ′(h) is the Gateaux derivative of F , and ‖ · ‖op is the operator norm ‖A‖op = inf{c > 0 : ‖Ax‖ ≤ c‖x‖∀x ∈ X}.
Proof. Consider G(η) = F (g + η(f − g)). Apply the ordinary mean value theorem to obtain
G(1)−G(0) = G′(c), c ∈ [0, 1] = F ′(g + c(f − g)) · (f − g)
and thus
|F (f)− F (g)| ≤ ‖F ′(h)‖op‖f − g‖
Claim 1. Consider P∞ = {pf = exp2−α(f − Aα(f)) : f ∈ L∞(S)}. Then for pf ∈ P∞, Aα(f) ≤ ‖f‖∞.
Proof.
pf (t) = exp2−α(f(t)−Aα(f)) ≤ exp2−α(‖f‖∞ −Aα(f)) for 1 < α ≤ 2∫
S pf (t)dQ ≤ ∫ S exp2−α(‖f‖∞ −Aα(f))dQ
1 ≤ exp2−α(‖f‖∞ −Aα(f)) log2−α 1 ≤ ‖f‖∞ −Aα(f) Aα(f) ≤ ‖f‖∞
where for the second line recall that we assumed that throughout the paper 1 < α ≤ 2.
Lemma 4. ConsiderP∞ = {pf = exp2−α(f−Aα(f)) : f ∈ L∞(S)}. Then the Frechet derivative of Aα : L∞ → R exists. It is given by the map
A′(f)(g) = Ep̃2−αf (g(T ))
= ∫ p2−αf (t)g(t)dQ∫ p2−αf (t)dQ
Proof. This proof has several parts. We first derive the Gateaux differential of pf in a direction ψ ∈ L∞ and as it depends on the Gateaux differential of Aα(f) in that direction, we can rearrange terms to recover the latter. We then show that it exists for any f, ψ ∈ L∞. Next we show that the second Gateaux differential of Aα(f) exists, and use that along with a functional Taylor expansion to prove that the first Gateaux derivative is in fact a Frechet derivative.
In Martins et al. (2020) they show how to compute the gradient of Aα(θ) for the finite dimensional case: we extend this to the Gateaux differential. We start by computing the Gateaux differential of pf .
d
dη pf+ηψ(t) =
d
dη exp2−α(f(t) + ηψ(t)−Aα(f + ηψ))
= d dη [1 + (α− 1)(f(t) + ηψ(t)−Aα(f + ηψ))]1/(α−1)+
= [1 + (α− 1)(f(t) + ηψ(t)−Aα(f + ηψ))](2−α)/(α−1)+ ( ψ(t)− d
dη Aα(f + ηψ) ) = p2−αf+ηψ(t) ( ψ(t)− d
dη Aα(f + ηψ)
)
evaluating at η = 0 gives us
dp(f ;ψ)(t) = p2−αf (ψ(t) + dAα(f ;ψ))
Note that by claim 1 we have
pf+ηψ(t) = exp2−α(f(t) + ηψ(t)−Aα(f + ηψ(t)) ≤ exp2−α(2‖f‖∞ + 2η‖ψ‖∞) ≤ exp2−α(2(‖f‖∞ + ‖ψ‖∞))
We can thus apply the dominated convergence theorem to pull a derivative with respect to η under an integral. We can then recover the Gateaux diferential of Aα via
0 = d
dη ∣∣∣∣ η=0 ∫ pf+ηψ(t)dQ
= ∫ dp(f ;ψ)(t)dQ
= ∫ pf (t) 2−α(ψ(t)− dAα(f ;ψ))dQ
dAα(f ;ψ) = Ep̃2−αf (ψ(T ))
<∞
where the last line follows as ψ ∈ L∞. Thus the Gateaux derivative exists in L∞ directions. The derivative at f maps ψ :→ Ep̃2−αf (ψ(T )) i.e. A ′ α(f)(ψ) = Ep̃2−αf (ψ(T )). We need to show that this is a Frechet derivative. To do so, we will take take second derivatives of pf+ηψ(t) with respect to η in order to obtain second derivatives of Aα(f + ηψ) with respect to η. We will then construct a functional second order Taylor expansion. By showing that the second order terms converge sufficiently quickly, we will prove that the map ψ :→ Ep̃2−αf (ψ(T )) is a Frechet derivative.
d2
dη2 pf+ηψ(t) =
d
dη pf+ηψ(t)
2−α ( ψ(t)− d
dη Aα(f + ηψ) ) = ( d
dη pf+ηψ(t)
2−α )(
ψ(t)− d dη Aα(f + ηψ) ) − pf+ηψ(t)2−α d2
dη2 Aα(f + ηψ)
= (2− α)pf+ηψ(t)(ψ(t)− d
dη Aα(f + ηψ))
d
dη pf+ηψ(t)
− pf+ηψ(t)2−α d2
dη2 Aα(f + ηψ)
= (2− α)p3−2αf+ηψ(ψ(t)− d
dη Aα(f + ηψ))
2 − pf+ηψ(t)2−α d2
dη2 Aα(f + ηψ)
We need to show that we can again pull the second derivative under the integral. We already showed that we can pull the derivative under once (for the first derivative) and we now need to do it again. We need to show an integrable function that dominates pf+ηψ(t)2−α(ψ(t)− Ep̃2−αf+ηψψ(T )).
|pf+ηψ(t)2−α(ψ(t)− Ep̃2−αf+ηψψ(T ))| ≤ pf+ηψ(t) 2−α2‖ψ‖∞
≤ exp2−α(2(‖f‖∞ + ‖ψ‖∞))2‖ψ‖∞ which is in L1(Q). Now applying the dominated convergence theorem
0 =
∫ d2
d 2 pf+ ψ(t)dQ
= ∫ [ (2− α)p3−2αf+ ψ(ψ(t)− d
d Aα(f + ψ))
2 − pf+ ψ(t)2−α d2
d 2 Aα(f + ψ)
] dQ
and rearranging gives
d2
d 2 Aα(f + ψ) = (2− α)
∫ p3−2αf+ ψ(ψ(t)− d d Aα(f + ψ))
2dQ∫ pf+ ψ(t)2−αdQ
d2
d 2 Aα(f) ∣∣∣∣ =0 = (2− α) ∫ p3−2αf (ψ(t)− Ep̃2−αf [ψ(T )]) 2dQ∫ pf (t)2−αdQ
since f, ψ ∈ L∞. For the functional Taylor expansion, we have from Lemma 2
Aα(f + ψ) = Aα(f) +A ′ α(f)(ψ) +
1 2 A′′α(f + ψ)(ψ) 2
for some ∈ [0, 1]. We thus need to show that for ∈ [0, 1],
(2− α) 1 ‖ψ‖∞
∫ p3−2αf+ ψ(ψ(t)− Ep̃2−αf+ ψ [ψ(T )])
2dQ∫ pf+ ψ(t)2−αdQ ψ→0→ 0
It suffices to show that the numerator tends to 0 as ψ → 0.∣∣∣∣ 1‖ψ‖∞ (ψ(t)− Ep̃2−αf+ ψ [ψ(T )])2 ∣∣∣∣
= ∣∣∣∣∣ ψ(t)‖ψ‖∞ψ(t)− ψ(t)‖ψ‖∞ 2Ep̃2−αf+ ψ [ψ(T )] + Ep̃2−αf+ ψ [ψ(T )] ‖ψ‖∞ Ep̃2−αf+ ψ [ψ(T )] ∣∣∣∣∣ ≤ ∣∣∣∣ ψ(t)‖ψ‖∞
∣∣∣∣ ∣∣∣ψ(t)− 2Ep̃2−αf+ ψ [ψ(T )]∣∣∣ +
∣∣∣∣Ep̃2−αf+ ψ ψ(T )‖ψ‖∞ ∣∣∣∣ ∣∣∣Ep̃2−αf+ ψ [ψ(T )]∣∣∣
≤ ∣∣∣ψ(t)− 2Ep̃2−αf+ ψ [ψ(T )]∣∣∣+ ‖pf+ ψ‖2−α2−α ∣∣∣Ep̃2−αf+ ψ [ψ(T )]∣∣∣
→ 0 as ψ → 0
and plugging this in we obtain the desired result. Thus the Frechet derivative of Aα(f) exists.
Lemma 5. Define P∞ = {pf = exp2−α(f −Aα(f)) : f ∈ L∞(S)} where L∞(S) is the space of almost surely bounded measurable functions with domain S. Fix f0 ∈ L∞. Then for any fixed > 0 and pg, pf ∈ P∞ such that f, g ∈ B ∞ (f0) the L
∞ closed ball around f0, there exists constant Mexp > 0 depending only on f0 such that
‖pf − pg‖Lr ≤ 2Mexp‖f − g‖∞

heading: Further
text: BΩα(pf , pg) ≤ 1
α− 1 ‖pf − pg‖∞[(‖pf‖∞ + ‖pf − pg‖∞)α−1 + exp2−α(2‖g‖∞)]
Proof. This Lemma mirrors Lemma A.1 in Sriperumbudur et al. (2017), but the proof is very different as they rely on the property that exp(x + y) = exp(x) exp(y), which does not hold for β-exponentials. We thus had to strengthen the assumption to include that f and g lie in a closed ball, and then use the functional mean value theorem Lemma 3 as the main technique to achieve our result.
Consider that by functional mean value inequality,
‖pf − pg‖Lr = ‖ expβ(f −Aα(f))− expβ(g −Aα(g))‖Lr ≤ ‖ expβ(h−Aα(h))2−α‖∞(‖f − g‖∞ + |Aα(f)−Aα(g)|) (13)
where h = cf + (1− c)g for some c ∈ [0, 1]. We need to bound expβ(h− Aα(h)) and ‖Aα(f)− Aα(g)‖∞.
We can show a bound on ‖h‖∞ ‖h‖∞ = ‖cf + (1− c)g − f0 + f0‖∞
≤ ‖c(f − f0) + (1− c)(g − f0) + f0‖∞ ≤ c‖f − f0‖∞ + (1− c)‖g − f0‖∞ + ‖f0‖∞ ≤ + ‖f0‖∞
so that h is bounded. Now we previously showed in claim 1 that |Aα(h)| ≤ ‖h‖∞ ≤ + ‖f0‖∞. Since h,Aα(h) are both bounded expβ(h−Aα(h))2−α is also. Now note that by Lemma 3,
|Aα(f)−Aα(g)| ≤ ‖A′α(h)‖op‖f − g‖∞ We need to show that ‖A′α(h)‖op is bounded for f, g ∈ B (f0). Note that in Lemma 4 we showed that
|A′α(f)(g)| = |Ep2−αf [g(T )]|
≤ ‖g‖∞ Thus ‖A′α‖op = sup{|A′α(h)(m)| : ‖m‖∞ = 1} ≤ 1. LetMexp be the bound on expβ(h−Aα(h)). Then putting everything together we have the desired result
‖pf − pg‖Lr ≤ 2Mexp‖f − g‖∞
Now
BΩα(pf , pg) = Ωα(pf )− Ωα(pg)− 〈∇Ωα(pg), pf − pg〉2 (14) For the inner prodct term, first note that following Martins et al. (2020) the gradient is given by
∇Ωα(pg)(t) = pg(t)
α−1
α− 1 (15)
Thus
|〈∇Ωα(pg), pf − pg〉2| ≤ ‖∇Ωα(pg)‖1‖pf − pg‖∞
= 1
α− 1 ∫ S exp2−α(g(t)−A(g))dQ‖pf − pg‖∞
≤ 1 α− 1 exp2−α(2‖g‖∞)‖pf − pg‖∞
where the second line follows from claim 1.
Further note that by Taylor’s theorem,
yα = xα + αzα−1(y − x) (16) for some z between x and y. Then letting y = pf (t) and x = pg(t), we have for some z = h(t) lying between pf (t) and pg(t) that
pf (t) α = pg(t) α + αh(t)α−1(pf (t)− pg(t)) Since f ∈ L∞ then applying Claim 1 we have that each pf , pg ∈ L∞ and thus h is. Then
|pf (t)α − pg(t)α| = α|h(t)|α−1|pf (t)− pg(t)| ≤ α‖h‖α−1∞ ‖pf − pg‖∞ ≤ αmax{‖pf‖∞, ‖pg‖∞}α−1‖pf − pg‖∞ ≤ α(‖pf‖∞ + ‖pf − pg‖∞)α−1‖pf − pg‖∞ (17)
so that
|Ωα(pf )− Ωα(pg)| = ∣∣∣∣ 1α(α− 1) ∫ (pf (t) α − pg(t)α)dQ ∣∣∣∣
≤ 1 α− 1 (‖pf‖∞ + ‖pf − pg‖∞)α−1‖pf − pg‖∞.
Putting it all together we obtain
BΩα(pf , pg) ≤ 1
α− 1 (‖pf‖∞ + ‖pf − pg‖∞)α−1‖pf − pg‖∞
+ 1
α− 1 exp2−α(2‖g‖∞)‖pf − pg‖∞
= 1
α− 1 ‖pf − pg‖∞[(‖pf‖∞ + ‖pf − pg‖∞)α−1 + exp2−α(2‖g‖∞)]

heading: C UWAVE EXPERIMENTS: ADDITIONAL DETAILS
text: We experiment with N = 64, 128 and 256 basis functions, and use a learning rate of 1e− 4. We use H = 100 attention mechanisms, or heads. Unlike Vaswani et al. (2017), our use of multiple heads is slightly different as we use the same value function for each head, and only vary the attention densities. Additional architectural details are given below.
C.1 VALUE FUNCTION
The value function uses regularized linear regression on the original time series observed at random observation times (which are not dependent on the data) to obtain an approximation V (t; B) = BΨ(t) ≈ X(t). The H in Eqn. 6 is the original time series.

heading: C.1.1 ENCODER
text: In the encoder, we use the value function to interpolate the irregularly sampled time series at the original points. This is then passed through a convolutional layer with 4 filters and filter size 5 followed by a max pooling layer with pool size 2. This is followed by one hidden layer with 256
units and an output v of size 256. The attention densities for each head h = 1, · · · , H are then
µh = w T h,1v
σh = softplus(wTh,2v)
γh = W (h)v
for vectors wh,1, wh,2 and matrices Wh and heads h = 1, · · · , H

heading: C.1.2 ATTENTION MECHANISM
text: After forming densities and normalizing, we have densities p1(t), · · · , pH(t), which we use to compute context scalars
ch = Eph [V (T )]
We compute these expectations using numerical integration to compute basis function expectations Eph [ψn(T )] and a parametrized value function V (t) = Bψ(t) as described in section 3.

heading: C.1.3 CLASSIFIER
text: The classifier takes as input the concatenated context scalars as a vector. A linear layer is then followed by a softmax activation to output class probabilities.

heading: D MIT BIH: ADDITIONAL DETAILS
text: Note that our architecture takes some inspiration for the H that we use in our value function from a github repository5, although they used tensorflow and we implemented our method in pytorch.
D.1 VALUE FUNCTION
The value function regresses the output of repeated convolutional and max pool layers on basis functions, where the original time series was the input to these convolutional/max pooling layers. All max pool layers have pool size 2. There are multiple sets of two convolutional layers followed by a max pooling layer. The first set of convolutional layers has 16 filters and filter size 5. The second and third each have 32 filters of size 3. The fourth has one with 32 filters and one with 256, each of size 3. The final output has 256 dimensions of length 23. This is then used as our H matrix in Eqn 6.

heading: D.2 ENCODER
text: The encoder takes the original time series as input. It has one hidden layer with a ReLU activation function and 64 hidden units. It outputs the attention density parameters.

heading: D.3 ATTENTION MECHANISM
text: The attention mechanism takes the parameters from the encoder and forms an attention density. It then computes
c = Ep[V (T )] (18)
for input to the classifier.

heading: D.4 CLASSIFIER
text: The classifier has two hidden layers with ReLU activation and outputs class probabilities. Each hidden layer has 64 hidden units.
5https://github.com/CVxTz/ECG_Heartbeat_Classification

Review summaries:
Many modern neural architectures, especially in natural language processing, rely heavily on the attention mechanism. Previous work in the literature proposed to extend the softmax-based attention mechanism by using different distribution families. In particular, the authors of this paper focus on variants of the attention mechanism that allow for continuous and sparse attention.  The authors propose kernel deformed exponential families, an extension of the exponential family that allows sparse and multimodal attention, contrary to most of previous work that focused on unimodal attention.

The manuscript considers an extension of the attention mechanism framework developed by others in recent work (by Martins et al.). Specifically this framework allows to break free from the discrete nature of attention that typically consists of a weighted average of a finite set of vectors. This is realized by estimating a probability mass function (PMF) over the finite collection of vectors, and then computing the expected value. The generalization allows one to extend the finite collection to a continuum. This is then handled by using a probability distribution function (pdf) over the collection to compute an expected value. In recent work, various authors considered the probability distribution used in defining the attention mechanism to belong to either a unimodal exponential family, a "deformed" exponential family (deformed versions having possibly finite support), or a mixture of Gaussians. Traditional exponential family comprises pdfs that possess a finite set of sufficient statistics. In contrast, the kernel exponential family comprises pdfs that have essentially infinitely many sufficient statistics through the use of a kernel. The current manuscript proposes to employ a kernel exponential family, and a deformed kernel exponential family. This allows them to work with multimodal probability distributions, and/or distributions with compact support. The authors layout conditions under which the kernel versions of the (deformed) exponential family are defined. They also apply the new attention schemes to several datasets.

The paper studies the continuous attention mechanism using the kernel exponential family and its deformed variant. This is an extension of existing works based on finite-dimensional exponential families. The authors investigated some theoretical conditions such that the RKHS defines the probability density functions. Numerical experiments showed that the proposed method works efficiently for some datasets.