Kernel Multimodal Continuous Attention
Authors: Alexander Moreno
Zhenke Wu
Supriya Nagesh
Walter H. Dempsey
James Matthew Rehg
Venue: NeurIPS 2022 Accept (2022)
Keywords: attention
continuous attention
kernel methods

Abstract: Attention mechanisms take an expectation of a data representation with respect to probability weights. Recently, (Martins et al. 2020, 2021) proposed continuous attention mechanisms, focusing on unimodal attention densities from the exponential and deformed exponential families: the latter has sparse support. (Farinhas et al 2021) extended this to to multimodality via Gaussian mixture attention densities. In this paper, we extend this to kernel exponential families (Canu and Smola 2006) and our new sparse counterpart, kernel deformed exponential families. Theoretically, we show new existence results for both kernel exponential and deformed exponential families, and that the deformed case has similar approximation capabilities to kernel exponential families. Lacking closed form expressions for the context vector, we use numerical integration: we show exponential convergence for both kernel exponential and deformed exponential families. Experiments show that kernel continuous attention often outperforms unimodal continuous attention, and the sparse variant tends to highlight peaks of time series.

Kernel Multimodal Continuous Attention
Alexander Moreno"
Zhenke Wu University of Michigan
Supriya Nagesh Georgia Tech
Walter Dempsey University of Michigan
Luminous Computing
James M. Rehg Georgia Tech
Abstract
Attention mechanisms average data representation with respect to probability weights. Recently, (23/25) proposed continuous attention, focusing on unimodal exponential and deformed exponential family attention densities: the latter have sparse support. g extended to multimodality via Gaussian mixture attention densities. In this paper, we propose using kernel exponential families [ and our new sparse counterpart, kernel deformed exponential families. Theoretically, we show new existence results for both families, and approximation capabilities for the deformed case. Lacking closed form expressions for the context vector, we use numerical integration: we prove exponential convergence for both families. Experiments show that kernel continuous attention often outperforms unimodal continuous attention, and the sparse variant tends highlight time series peaks.
Introduction
Attention mechanisms are weighted averages of data representations used make predictions. Discrete attention 1) cannot easily handle irregularly spaced observations, and 2) attention maps may be scattered, lacking focus. [23 24 proposed continuous attention, showing that attention densities maximize the regularized expectation of function of the data location (i.e. time). Special cases lead to exponential and deformed exponential families: the latter has sparse support. They form continuous data representation and take expectations with respect to attention densities. In [25] they apply this to a transformer architecture.
23 used unimodal attention densities, giving importance to one data region. multimodal Gaussian mixture attention densities. However 1) Gaussian mixtures lie in neither the exponential nor deformed exponential families, and are difficult to study in the context of [23] 24]: and 2) they have dense support. Sparse support can say that certain regions of data do not matter: region of time has no effect on class probabilities, or region of an image is not some object. We would like to use multimodal exponential and deformed exponential family attention densities, and understand how g relates to the framework of [23] 24).
g extended this to
This paper makes three contributions: 1) introduce kernel deformed exponential families, a sparse multimodal density class, and apply it along with the multimodal kernel exponential families A attention densities. The latter have been used for density estimation, but not weighting data importance; 2) we theoretically analyze kernel exponential and deformed exponential family i) normalization, ii) approximation and iii) context vector numerical integration properties; 3) we apply
An earlier draft was done at Georgia Tech as a dissertation chapter. Additional experiments and extensive rewriting were done at Luminous Computing.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

them to real world datasets, showing that multimodal continuous attention outperforms unimodal, and that kernel deformed exponential family densities often highlight the peaks of time series. Approximation properties for the kernel deformed case are challenging: similar kernel exponential family results [33] relied on exponential and logarithm properties to bound the difference of the log- log-partition functional at two functions: these do not hold for deformed analogues. We provide similar bounds by using a mean value inequality and bounding a functional derivative.
We first review unimodal continuous attention [23 24 We motivate multimodal continuous attention via time warping. We next describe kernel exponential families and give a novel normalization condition relating kernel growth to the base density's 's tail decay. We then propose kernel deformed exponential families, new densities with support over potentially disjoint regions. We describe normalization and approximation capabilities. Next we describe using these densities for continuous attention, including numerical integration convergence analysis. We show experiments comparing unimodal and multimodal attention, and conclude with limitations and future work.
2 Related Work
Attention Mechanisms closely related are 23 8
frame continuous attention as an expectation of value function with respect to density, where the density solves an optimization problem. They only used unimodal (deformed) exponential family densities: we extend this to the multimodal setting by leveraging kernel exponential families and proposing a deformed counterpart. 8 proposed a multimodal continuous attention mechanism via a Gaussian mixture. We show in Appendix [ that this solves slightly different optimization problem from P 24 A limitation of Gaussian mixtures is lack of flexible tail decay. Finally, (25] apply continuous attention within transformer architecture to model long context. This is application of continuous attention rather than an extension of specific continuous attention mechanisms.
Also relevant are 40 30 31. 30 provide an attention mechanism for irregularly sampled time series by use of continuous time kernel regression framework, but do not take an expectation of a data representation over time with respect to continuous pdf. Instead they evaluate the kernel regression model at fixed time points. This describes importance of data at a set of points rather than over continuous regions. 31 extend this to incorporate uncertainty quantification. Other papers connect attention and kernels, but focus on discrete attention 440 5 Also relevant are temporal
transformer papers, including [45 15
17 32
However, none have continuous attention densities.
Kernel Exponential Families 4 proposed kernel exponential families: [33 analyzed theory for
density estimation. 44] parametrized the kernel with a deep neural network. Other density estimation
papers include [ 6 36). We apply kernel exponential families attention densities to weight a value function which represents the data, rather than for density estimation. Further, [4] showed a condition for an unnormalized kernel exponential family density have a finite normalizer. However, they used exponential power base densities. We instead relate kernel growth rates to the base density tail decay, allowing non-symmetric base densities.
To summarize our theoretical contributions: 1) showing that multimodal continuous attention is required to represent time warping 2) introducing kernel deformed exponential families with approxi- mation and normalization analysis 3) improved kernel exponential family normalization results 4) stability and convergence analysis of numerical integration for kernel-based continuous attention 5)
characterizing g in terms of the framework of [23]
3
Continuous Attention Mechanisms
An attention mechanism has: 1) value function: a raw or learned data representation 2) an attention density chosen to be 'similar' to another data representation, encoding it into a density 3) context c 23 taking an expectation of the value function with respect to the attention density:
c= Er (T))
The value function V: R a data representation, T ~ p(() is the random variable or vector for locations (temporal, spatial, etc) in domain S, and p(t) is the attention density (potentially with respect to a discrete measure). For discrete attention, one could have (t)) be a time series where t E S finite set of time points. One then weights the time series with probability vector to

---

obtain the context vector. An example of this for irregularly sampled time series is [30.. For each attention mechanism (eqn. in their paper), S is the set of observed time points, the value function maps the observed time points to their respective observation values, and the attention probability mass function is the output of normalized kernel evaluations between observation time points and reference time point.
In the continuous setting (tt) could be curve or realization of continuous-tine stochastic process, and S could be [0, where is a study end time. One then weights it with a continuous probability density. If is a of spatial locations and one has image data, then the value function could be raw image or learned representation of an image. Finally, for self attentonn [18 42), V is linear transformation of a sequence, and the expectation is conditional on transformation of a specific
token (query).
To choose p, one takes data representation and finds p 'similar' to f. but regularizing p. 23 24
did this, formalizing attention mechanisms. Given a measure space (S, Q), let .M (S) be the set of probability densities with respect to Q. Assume Q is dominated by a-finiie measure / (i.e. Lebesgue) and that it has Radon Nikodym derivative go dQ with respect v. Let S R F be function class, and Ω M (S) R be lower comi-continuous, proper, strictly convex functional.
Given f E F, an attention density (23) p: S - Rz0 solves
/ p(t))(t)dQ(()) S(p).
p arg max TEM)(8)
(2)
This maximizes regularized 1,2 similarity between p and a data representation ʃ. If s2(p) Ss p() (ogr(())Q(() is the negative differential entropy, the attention density is Boltzmann Gibbs
=
f((()) exp(f(()- A(f)),
where A(f) ensures Ss (lf)(t)dQ (see 23 for proof). If f(t) = 6T 6(t) for parameters and statistics 0 E R-M,, ei() E JM respectively, Eqn. becomes an exponential family density. For f in reproducing kernel Hilbert space (RKHS) H. it becomes a kernel exponential family density [4) which we propose as an alternative attention density.
One desirable class would be heavy or thin tailed exponential family like densities. In exponential families, the support, or non-zero region of the density, controlled by the measure Q. Letting (2(p)
be the a- Tsallii negative entropy S, (p) [41].
(p( = ola-1) (ssp(t) 1), a # 1;
p(() log p(l)dQ a= 1,
then plf] for f(t) = (tt) lies in the deformed exponential family 27
Po. (f((t) = exp;-a x()) - Af))),
(4)
where Af( again ensures normalization and the density uses the 3-exponential
[1 (1 3) 1/1-8) $) 1
e)p
(5)
exp(t), 3 1.
For 3 < 1, Eqn. 5|d thus deformed exponential family densities for < can return 0 values.
Values a > (and thus B < 1) give thinner tails than the exponential family, while a < 1 gives fatter tails. Setting sparse case studied in (23). We again propose to replace f(t) = 6" o(t) with f E H. which leads to the novel kernel deformed exponential families.
0 is called sparsemax (22). In this paper, we assume a < 2, which the
Computing Eqn. Is context vector requires parametrizing (t).. (23 parametrize V : - R with B RDX ((t: = B (t) and estimate B E R. 1>xx>x via regularized multivariate linear N regression. Here V = [ is a set of basis functions. Let L be the number of observation locations (times in a temporal setting). D be the observation dimension, and N be the number of basis functions. Using Frobenius norm || ||,, this involves regressing the observation matrix H E RDL on matrix F E N of basis functions (M, evaluated at observation locations (4)/
B arg min
BF H X B
3

---

4
Time Warping
We now draw connection to time warping [28] to show an advantage of our method. One summary statistic for classification is a global weighted average pooling: an expectation of temporal features. However in many processes features may not be aligned in time, and we only observe unaligned curves due to a discrepancy between individual system times and clock time: this is known as phase variation (2]] For instance, electrocardiogram (ECG) heartbeat curves have P-wave, QRS complex and a T-wave. These have similar patterns between heartbeats, but may have different durations and peak locations. Here we show that the expectation of temporally aligned curve with respect to global density is equivalent to the expectation of the unaligned curve with respect to an individualized density. However even if the global density is unimodal, the individualized density may not be. We first define the function that aligns a set of features to common reference times. Definition 4.1. (Time Warping Function) Given references times {fox kk_ and individualized times {tue ) both in [], time warping function h4 S + R for S C Rzo is strictly increasing, differentiable, invertible function where
h0) 0, (T) =T
(t(t) = tix, k = 1....,K
ha(t) f i & [0,-]
Let {X X;: S S-
+ R be observed curves, each with K features occurring at individualized 10, increasing in k. A set of time warping functions (h: a map reference [e= C
times (fa k=]
times to individualized feature times. One can then compute aligned X (t) = (t)) Each X² has relevant features at the same times {lo*] Classically, this requires handcraftngg and locating important features and estimating a warping function. We could then compute an expectation of the time warped curve with respect to global fixed density p(t) to obtain summary statistic ET (T) of the aligned curve. The following states that multimodal continuous attention can represent such an expectation with an attention density Ps, avoiding computing (t).
Lemma 4.2. (Continuous Attention can Represent Time Warping) Let h be a time warping function, = h and X : IR + IR with support on [0, T. Assume that Q is dominated by Lebesgue measure v dQ,
and let qu we have du- Then for any fixed density p wrt Q, if 9, X3, 9o,P are continuous almost everywhere
Bu-p (U)
ET (!(T)
where (t()) = p():(t)) (t)) g)()) and (t) is a valid probability density function. (o(()
See Appendix B.I for proof. Even if p(t) is unimodal, (t) may not be: see Appendix B.2 for an example. Thus we require multimodal continuous attention to represent such statistics.
5 Kernel Exponential and Deformed Exponential Families
We use kernel exponential families and a new deformed counterpart obtain flexible attention densities solving Eqn. 2 with the same regularizers. We first review kernel exponential families. We then give a novel theoretical result describing when an unnormalized kernel exponential family density can be normalized. This says that the normalizing constant exists when the base density has fast enough tail decay relative to kernel growth. This result allows us to verify that a choice of base measure and kernel lead to valid attention density and thus attention mechanism. Next we introduce kernel deformed exponential families, extending kernel exponential families to have either sparse support, our focus, or fatter tails. These can attend to non-overlapping time intervals. We show similar normalization results based on kernel choice and base density. The normalizing constant exists when the unnormalized density has compact support and the kernel grows sufficiently slowly, Following this we show approximation theory. We conclude by showing how to compute attention densities in practice.
Kernel exponential families + extend exponential families, replacing f(t) = 0T (()( with f f H. reproducing kernel Hilbert space H [2] Densities can be written
p()) = exp(J(()= - A(J))
= cf k(:()))) A(f)).
(8)

---

FordA, Original Scries vs Roscaled Attortion
ECG. Origina Series
Spasoree Rorve
Bonel sparserr
34
Figure I: Rescaled continuous attention densities for kernel deformed exponential families engine noise example b) kernel deformed exponential families on an ECG example. Both examples are multimodal and highlight salient features of signal, and the ECG example in particular highlights the waves, which describe the electrical signal passing through the heart conduction system. It first highlights the R wave, the largest peak, then the T wave, and next the wave again.
Eqn. [8 follows from the reproducing property. challenge to choose H. Q that normalizing constant exists, i.e., Ig exp(f(t))dQ < co. These densities can approximate any continuous density over compact domain arbitrarily well in KI. divergence, Hellinger, and T.P distance 333. However relevant integrals including the normalizing constant require numerical integration.
To avoid infinite dimensionality one generally assumes representation of the form where for density estimation [33] the t, are the observation locations and this is the solution to a regularized empirical risk minimization problem. This requires using one parame- ter per observation value. This model complexity may not be necessary, and often one chooses set of inducing points (38] (4.)/z where 7 is less than the number of observation locations.
For given pair H. how can we choose to ensure that the normalization constant exists? We first give a simple example of H. f and Q where it does not. Example 1. Let Q be the law of (0. 1) distribution and S = R. Let H = span (2) with k(1, 8) = 3 + g1 and f()) = 1 + 1 k(t,, 1). Then the following integral diverges.
[ exp(f(t))dQ J v 2m
x2 (-3
exp
3 + +
dt
2
5.1 Theory for Kernel Exponential Families
We provide sufficient conditions for Q and H so that A(f) the log- aeg-paritionn functional exists. We relate H's kernel growth rate the tail decay of the random variable or vector To with law Q. Proposition 5.1. Let p(t)= = ct)(f(t)) where f H an RKHS with kernel k. Assume k(t, t) I Lellt 222 + Cj for constants LR. CR. > 0. Let Q be the law of random vector TQ. that ((A) = P(To E A). Assume Vu s.1. ||u||2 = 1, > 0
P( 2 x) S C2 exp(-uz")
(9)
for some constants Cq. 0. Then
[ p(t)dQ < x0.
vary what tail decay rate for To ensures we can normalize p((). [4 also proved normalization conditions, but focused on exponential power density for a specific growth rate of k(t, t) rather than relating tail decay to growth rate. By focusing on tail decay, our result can be applied to non-symmetric base densities. Specific kernel bound growth rate terms lead allowing different tail decay rates.
See Appendix C. for proof. Based k((, growth,
Corollary 5.2. For & 4, To can be any sub-Gaut Gaussian random vector. For & it can be any sub-exponential. For & = it can be any probability density.
See Appendix C.2 for proof.
5

---

5.2 Kernel Deformed Exponential Families
We now propose kernel deformed exponential families, which are flexible sparse non-parann non-parametric densities: these can be multimodal. They take deformed exponential families and extend them to use kernels in the deformed exponential term. This mirrors kernel exponential families. We write
p(() exp2-a(f(1) Aa()))
where f E H. with kernel k. Fig. shows that they can have support over disjoint intervals.
5.2.1 Normalization Theory
We construct valid kernel deformed exponential family density from Q and f E H. We first discuss the deformed log- normalizer. exponential family densities, the log-normalizer is the log of the normalizer. For deformed exponentials, the following holds.
Lemma 5.3. Let > 0 be a constant. Then for < a 2,
exp2-alz"- f(t)) = exp2-a logg 2)
Z
where
ا-مーي > > 0,3 x I-3
log,
log(t) ift > 0, = 1;
undefined ift < 0.
See Appendix D.I for proof. We now describe a normalization sufficient condition analagous to Proposition [.. [ deformed kernel exponential families. With Lemma [.3 and an unnormalized (ft derive valid normalized kernel deformed exponential family density. We only require that an affine function of the terms in the deformed-exponente is negative for large [||]2-
Proposition 5.4. For < S 2 assume p(() exp2-a (F(1) with f E H. H is RKHS with kernel k. If, > s.t. for [|||[2 > Co - 1) S 0 and k(t, t) S La L for some & <>0, 0, then Ig (f < co.
See Appendix D.2 for proof. We now construct valid kernel deformed exponential family density using the finite integral.
Corollary 5.5. Under proposition 4 conditions, assume exp2 (f(t)) > on set A C S where
Q(A) >> 0, then constants Z 0, A. (J) R such that for ((I) z f((), the following holds
/
exxp expp2aa((()) Aa(f)dQQ
See Appendix D.3 for proof. We thus estimate f(t) fa=1f(t) and normalize to obtain density of the desired form.
5.2.2 Approximation Theory
Kernel deformed exponential families can approximate continuous densities satisfying tail condition on compact domains arbitrarily well in L2 norm, Hellinger distance, and Bregman divergence.
Theorem 5.6. Let go C(S), such that (t()) > for all E S, where C Rd is locally compact Hausdorff and qo is the Radon Nikodym derivative of measure Q with respect to a dominating measure
Suppose there exists I such that for any 0, 3R > 0 satisfying (p(l) [| S e for any with ||t||2 > R. Define
P. - (c=(pEC(S) ( ((())Q=1,pl
2 0, Vt S and p p- lE Co(S))
Suppose k(I,.) E Co(S)VL S and the kernel integration condition (Eqn. 12 holds. Then kernel deformed exponential families are dense Pe wrt T. norm, Hellinger distance and Bregman divergence for the a-Tsallis negative entropy functional.

---

The proof (Appendix D.4) idea is that under a kernel integrability condition, deformed exponential families parametrized by f H. are dense in those parametrized by Co( (we denote those parametrized by f Co(S) as Po). We can approximate C(S) densities satisfying the tail condition with Po densities, and thus with deformed exponential family densities. This extends [33]ss 's approximation to the deformed case: standard log and exponential rules cannot be applied. It requires bounding functional derivatives and applying a mean value inequality.
5.3
Using Kernels for Continuous Attention
Here describe how to compute continuous attention mechanisms with attention densities parametrized by functions in an RKHS H in practice. Algorithm [ shows the kernel deformed exponential family case: the kernel exponential family case involves a similar algorithm. Given base measure, kernel, and inducing point locations, we start by computing kernel weights % for fik(t, f(t) Zo-1f(( = Ek=ttkk(t, tz) and estimating the matrix B for basis weights for the value function (tt) (t). Unlike density estimation, this form for f simply a practical way to obtain I in an RKHS, rather than a solution to an empirical risk minimization problem. We then compute the normalizing constant Z SS exp2-o (f())) via numerical integration and use it to normalize f(t) to obtain the attention density p(t). Finally we compute the context = Er~pIV [V))] BE,, (t)) by taking the expectation of L (T) with respect to a deformed kernel exponential family density p.
Unlike [23 24). we lack closed form expressions and use numerical integration. In the backwards pass we use automatic differentiation. Note that in some cases we have numerical underflow when computing the normalizing constant. We also found that using FP16 precision, the default on some newer GPUs for Pytorch up to 1.11, leads to worse performance, sometimes dramatically so, and thus recommend using FP32. We discuss solutions for the underflow issue in Appendix E.
Algorithm 1 Continuous Attention Mechanism via Kernel Deformed Exponential Families
Choose (t) and kernel k. Inducing point locations {
Parameters {% the weights for f() (Z)a-1 f(t) E/n kkt(t, t,), matrix B for basis
weights for value function (I))= (1). ). I number of inducing points.
Forward Pass
Compute = exp2-o ( (()))((())))))))))))))))))) to obtain p()) = 1/2 exp2-a via numerical integration
Compute Er [x (T)) via numerical integration
Compute e c= Er-pl [V (T) = (T))
Backwards Pass use automatic differentiation
5.3.1 Numerical Integration Convergence
The trapezoidal rule's standard one-dimensional convergence rate is O( NZ for an integral over a fixed interval, where N is the number of grid points. We would like better convergence guarantees. We can achieve exponential convergence for the numerical integrals of kernel exponential and deformed exponential family attention. We focus on numerical integration over the real line, leaving truncation analysis and higher dimensions to future work. We let h > 0 be the grid size.
For functions holomorphic in strip with rapid decay, the trapezoidal rule has exponential convergence. For kernel exponential family attention, this gives us ((ex((-Chh) or exponential convergence for some C 0 with appropriate choice of qo- V. and k. Technical details are E.2, and are based on extending real-valued analytic functions to complex functions analytic/holomorpicc on a strip.
Kernel deformed exponential families, however, are not even differentiable, but we can construct a sequence of differentiable approximations by replacing the positive part/ReLU function in the deformed exponential with softplus for increasing values of the softplus parameter. Each differentiable approximation has exponential convergence, and by taking limits as the softplus parameter tends to infinity we can show that the numerical integral for kernel deformed exponential family attention itself has exponential convergence. Technical details are in E.3
We also show empirical convergence analysis in Appendix E.4 and figure 2 in that appendix. Both kernel exponential and deformed exponential families see rapid convergence for 1d attention, pro-
continuuus function on domain S vanishing at infinity
7

---

Attention
Accuracy
Cts Softmax
17.22+$4.20
Cts Sparsemax
77.96
Kernel Softmax (ours)
85.71 + 11.98
Kernel Sparsemax (ours)
88.1 1.50
Table 1: Results for 100 runs each of synthetic time warping classification experiment, N = 64. This involves generating 10, 000 trajectories, each of length 95 of unaligned curves.
viding excellent integral approximations with only 5-10 grid points. Further, the numerical integral using softplus is very close approximation to that using ReLU for softplus parameters and 10.
6 Experiments
We investigate how often multimodal continuous attention outperforms unimodal, given the same architecture. We also investigate whether these methods learn rich multimodal densties. We denote kernel exponential family attention as kernel softmax and the deformed case as kernel sparsemax. Our architectures have: 1) an encoder maps a discrete time series representation to attention density parameters. 2) The value function (t; B) expresses an embedding of a time series as a linear transformation of basis functions. 3) The context = (T))( which is used in 4) a classifier. Fig. Biin the Appendices visualizes this. For kernel softmax/sparsemaxx the encoder outputs are the weights for the kernel evaluations. For the Gaussian mixture case, the encoder's outputs are mixture weights and components means and variances. Here we describe one synthetic and three real data experiments. In Appendix [F] we provide an additional ECG classification experiment. We provide confidence intervals for three smaller experiments (at most 10, 000 samples). Our UWave and ECG experiments were done on a Titan X GPU, IMDB on a 1080, and FordA on an A40. We found that the A40 provides very different results out of the box for both accuracy/F1 and attention densities. As an example, Figure S was done on a Titan X with an older version of Pytorch, while Figure 7 was done with an A40 with Pytorch 1.12. Code is in our repository where discuss the flags used to control precision on recent GPUs and Pytorch versions. We discuss computational/ memory complexity in Appendix F and give summary in Table 5 We provide wall clock times in Table 5
6.1 Synthetic Experiment: Time Warping
We simulate time warping and do prediction for unimodal multimodal attention densities. Details of the original time aligned stochastic process and inverse warping function are in Appendix [B.]] Given
global densities for classes Pl 122 and aligned X*, the class is arg max((p1, X*), (p2, X We generate 10, 000 trajectories of length 95 observed at evenly spaced time points the interval [o, 1].
We use two attention mechanisms (heads), one for each class. Letting V be the value function fit to
observed data, the classifier is softmax( ((p.. V) (Pt2 L=]) where P:i and p12 are the attention densities for the first and second class. 1.96 standard deviation intervals. Kernel methods outperform by - 10%.
prediction results from 100 runs along with
6.2 FordA Dataset
This is binary classification dataset for whether a sympton exists in an automotive subsystem. Each time series has 500 sensor observations, and there are 3601 training samples and 1320 test samples. 2 results. Kernel sparsemax outperforms most baselines") while kernel softmax also does well. We use the same methods and architecture as the previous section, although hyperparameters slightly different. Several methods, including discrete softmax, continuous sparsemax and the transformer, have very poor performance. To sanity check we fit SVM with a Gaussian kernel, logistic regression, and decision tree: accuracies are under 55% for these, and we conclude that
"A earlier pre- pr-print of this paper using previous T'AA//yYTorc versions and a Titan X instead of an A40 had ~ 90% accuracy/F1 for LSTM FCN. We no longer have access to that environment, but we suspect the difference has do with either precision or implementation changes in the TSAI library.
8

---

Method
Accuracy
F1
Discrete Softmax
51.27 + 1.86
33.89 0.82
Cis Softmax
74.88 + 21.36
+
Cts Sparsemax
90.95 + 1.77
00.94=1.77
Gaussian Mixture
69.56 35.94
52.88
Kernel Softmax (ours)
92.44 + 1.96
92.43 2.00
Kernel Sparsemax (ours)
92.61 + 1.62
92.60 + 1.62
LSTM FCN
93.41 + 0.43
93.40 0.43
TST (Transformer)
49.48 + 1.04
49.46 1.02
Table 2: FordA accuracy, 1.96 SD intervals over 10 runs. Our methods outperform most baselines.
Several methods have very poor performance. To sanity check we fit three classical methods: kernel SVM, logistic regression, and decision tree. All have under 55% accuracy, suggesting that this dataset is problematic for some methods.
Attention
N=64
N=128
Cts Softmax
6..78=1.64
67.70= 2.49
Cts Sparsemax
74.20+22.72
74.69:13.78
Gaussian Mixture
81.13 1.76
80.99:12.79
Kernel Softmax (ours)
93.85
0.60
94.26
Kernel Sparsemax (ours)
92.32 + 1.09
92.15 + 0.79
Table 3: Accuracy results on wave gesture classification dataset for the irregularly sampled case. Again over 10 runs. Due to the irregular sampling, this is only comparable to [16] 29 Kernel based attention substantially outperforms unimodal and mixture models. All methods use 100 attention heads. Gaussian mixture uses 100 components (and thus 300 parameters per head), and kernel methods use 256 inducing points.
this data poses difficulty for some classifiers. Appendix provides some additional details, along with attention density plots. The kernel softmax plots often highlight zero crossings. The kernel sparsemax plots often select peaks of a signal while learning rich sparsity patterns.
6.3 wave Experiment: Gesture Classification
We investigate: 1) Does a large number of unimodal attention heads, as suggested in (23 24], perform well when multimodality is needed? 2) Can this method work well for irregularly sampled time series? 3) Can we learn interesting multimodal attention densities?
We analyze Wave [9]: accelerometer time series with eight gesture classes. We follow 16 ss split into 3,582 training observations and 896 test observations: sequences have length 945. We do synthetic irregular sampling and uniformly sample 10% of the observations. Because of this our results are comparable to other uaave irregular sampling papers [16. 29], but not to results using the full time series.
Table 3 shows the results. Our highest accuracy 94.26%, the multi-head unimodal case's best is 74.69%, and the mixture's best is 81.13%. We outperform the results of [16]. who report highest accuracy of 91.41%. and perform similarly [29] (their figure suggests approximately 94% accuracy). Fig. shows attention densities for one of the attention heads for the first three classes. This takes one attention density for each time series of each class and plots it. Within the same class, all attention densities for the head (one for each time series) are plotted. The plot shows two things: firstly, attention densities have support over non-overlapping time intervals. This cannot done with Gaussian mixtures, and the intervals would be the same for each density in the exponential family case. Secondly, there high similarity of attention densities within each class, but low similarity between classes. Appendix describes additional details.

---

Attention
N=32
N-64
N=128
Mean
Cts Softmax
89.56
90.32
91.08
90.32
Cts Sparsemax
89.50
90.39
89.96
89.95
Kernel Softmax
91.30
91.08
90.44
90.94
Kernel Sparsemax
90.56
90.20
90.41
90.39
Table 4: IMDB sentiment classification dataset accuracy. Continuous softmax uses Gaussian attention, continuous sparsemax truncated parabola, and kernel softmax and sparsemax use kernel exponential and deformed exponential family with a Gaussian kernel. The latter has a = exponential and multiplication terms. N: basis functions, I 10 inducing points, bandwidth 0.01.
6.4 IMBD Sentiment Classification
We extend 23 's code for IMDB sentiment classification 20
This uses document representation from a convolutional network and an LSTM attention model. We use Gaussian base density and kernel, and divide the interval [0, 1] into I = 10 inducing points where we evaluate the kernel in (()) = E ra((t,4) 4:)- Table shows results. On average, kernel exponential and deformed exponential family slightly outperTorms the continuous softmax and sparsemax. The continuous softmax//parsemax results are from running their code.
Discussion
In this paper extend continuous attention mechanisms to use kernel exponential and deformed exponential family densities. The latter is a new flexible class of non-parametric densities with sparse support. We show novel existence properties for kernel exponential and deformed exponential families, prove approximation properties for the latter, and show exponential convergence of numerical integration for both attention mechanisms. We then apply these to the continuous attention framework described in [23 [44]. We show results on several datasets. Kernel attention mechanisms tend to outperform unimodal attention, sometimes by a large margin. We also see that in many cases they exhibit multimodality. Kernel sparsemax in particular learns rich sparsity patterns while highlighting peaks of signal. This was evident in Figure [ The engine noise example mimicked certain parts of the signal, while in the ECG example, it tends to give very high weight to R waves in a signal. While this paper more focused on general methods than a specific potentially dangerous upplication, potential application areas include wearable sensors and NLP, and general negative societal impacts those application areas could apply to this work.
7.1 Limitations
limitation the of numerical integration, which scales poorly with location dimensionality. While we achieve ID exponential convergence, we still must investigate whether we can extend this higher dimensions. This still allows for multiple observation dimensions given 1d location, i.e. multivariate time series or language tasks.
Funding and Other Acknowledgments
Alexander Moreno is supported by Luminous Computing, and was previously supported by NIH -PP41-EB002222-011A1 UO1CAA29437, and a Google CMD-IT Flip-Alliance fellowship. Zhenke Wu is partly supported by NIH U0ICA229437 and an investigator grant from Precision Health Initiative at University of Michigan, Ann Arbor. Supriya Nagesh is supported by NSF CNS1823201. Walter Dempsey is supported by P50 DA054039. We thank Marcos Treviso and André Martins for helpful discussions about their prior work and help with understanding their code.
23 repository for this dataset is http://giitubbcom//deepppppppppp
10

---

References
[1] Michael Arbel and Arthur Gretton. Kernel conditional exponential family. In International Conference on Artificial Intelligence and Statistics, pages 1337-1346. PMLR, 2018.
[2] Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American mathemati- cal society, 68(3):337-404, 1950.
[3] Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. 3rd International Conference on Learning Representations, ICLR 2015, 2015.
[4] Stéphane Canu and Alex Smola. Kernel methods and the exponential family. Neurocomputingg 69(7-9):714-720, 2006.
[5] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos. Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations, 2020.
[6] Bo Dai, Hanjun Dai, Arthur Gretton, Le Song, Dale Schuurmans, and Niao He. Kernel exponential family estimation via doubly dual embedding. In The 22nd International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2019.
[7] Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chis Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive. IEEE/CAA Journal of Automatica Sinica, 6(6)) 190111999-15,111-. 2019.
[8] António Farinhas, André FT Martins, and Pedro MQ Aguiar. Multimodal continuous visual attention mechanisms. In 202/ /EEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 1047-1056. IEEE, 2021.
[9] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung- Kang Peng, and Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101(23):e215-2220 2000.
[10] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[11] Robert Israel
israel). Determine if f(z) log(e" + 1) analytic and where. Mathematics Stack Exchange. URL / stackexchange com/ com/q/3060/00 2019-01-03).
(version:
[12] Mohammad Kachuee, Shayan Fazeli, and Majid Sarrafzadeh. Bcg heartbeat classification: A deep transferable representation. In 2018 IEEE International Conference on Healthcare Informatics (ICHI), pages 443 444. IEEE, 2018
[13] Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Shun Chen. Lstm fully convolutional networks for time series classification. IEEE access, 6:1662-1669, 2017.
[14] Yin Tat Lee. Lecture 19: Exponentially convergent trapezoidal rule. URL httts::///iit:at Son/teeaching/cse5589 witt99/1iittt11/111...1 pdf
[15] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In H. Wallach, Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, pages 5243-5253. Curran Associates, Inc., 2019. URL wttp::////poooeee...neee.eee wwwp:99eebbbbb-PPpeppeeee Paper.

---

[16] Steven Cheng-Xiun Xian Li and Benjamin M Marlin. A scalable end-to-end gaussian process adapter for irregularly sampled time series classification. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon. and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, pages 1804-1812. Curran Associates, Inc., 2016. URL http:ps://proceeanggg..... Paper. pdf.
[17] Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza for electronic health records. Scientific reports, 10(1):1-12, 2020.
Behrt: transformer
[18] Zhouhan Lin, Minwei Feng. Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In International Conference on Learning Representations, 2017.
[19] Jiayang Liu, Lin Zhong, Jehan Wickramasuriya, and Venu Vasudevan. uwave: Accelerometer- based personalized gesture recognition and its applications. Pervasive and Mobile Computing, 5(6):657-675, 2009.
[20] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150,
2011.
[21] James Stephen Marron, James Ramsay, Laura M Sangalli, and Anuj Srivastava. Functional data analysis of amplitude and phase variation. Statistical Science, pages 468-484, 2015.
[22] Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning, pages 1614 1623. PMLR, 2016.
[23] André Martins, António Farinhas, Marcos Treviso, Vlad Niculae, Pedro Aguiar, and Mario Figueiredo. Sparse and continuous attention mechanisms. In H. Larochelle, M. Ranzato, R. Had- sell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20989-21001 Curran Associates, Inc., 2020. URL http:: htttps::/pprcceednngg neurips. http:////ppppppo//o/////////ttta
Papere
[24] André FT Martins, Marcos Treviso, António Farinhas, Pedro MQ Aguiar, Mário AT Figueiredo, Mathieu Blondel, and Vlad Niculae. Sparse continuous distributions and fenchel-young losses. Journal of Machine Learning Research, 23:1-74, 2022.
[25] Pedro Henrique Martins, Zita Marinho, and André FT Martins. co-former: Infinite memory mansformer-formee Infinite memory transformer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2022.
[26] Sajad Mousavi and Fatemeh Afghah. Inter-and intra-patient eog heartbeat classification for arrhythmia detection: a sequence sequence deep learning approach. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1308-1312. IEEE, 2019.
[27] Jan Naudts. Estimators, escort probabilities, and-exponential families in statistical physics. JIPAM. Journal of Ineuulitiees in Pure & Applied Mathematics (electronic only], 5(4):Paper 2004.
[28] James O Ramsay and Xiaochun Li. Curve registration. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 60(2):351-363, 1998.
[29] Satya Narayan Shukla and Benjamin Marlin. peerpolation-prediitin networks for irregularly sampled time series. In International Conference on Learning Representations, 2019.
[30] Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2021. URL.
https://openrevier. htt.://wwww...eteooooooroooooood
12

---

[31] Satya Narayan Shukla and Benjamin M Marlin. Heteroscedastic temporal variational autoen- coder for irregularly sampled time series. International Conference on Learning Representations. 2022.
[32] Huan Song. Deepta Rajan, Jayaraman Thiagarajan, and Andreas Spanias. Attend and diagnose: Clinical time series analysis using attention models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[33] Bharath Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Aapo Hyvärinen, and Revant Kumar. Density estimation in infinite dimensional exponential families. The Journal of Machine Learning Research, 18(1):1830-188) 2017.
[34] Bharath K Sriperumbudur, Kenji Fukumizu, and Gert RG Lanckriet. Universality, characteristic kernels and rkhs embedding of measures. Journal of Machine Learning Research, 12(7), 2011.
[35] Elias M Stein and Rami Shakarchi. Complex analysis, volume 2. Princeton University Press, 2010.
[36] Dougal Sutherland, Heiko Strathmann, Michael Arbel, and Arthur Gretton. Efficient and princi- pled score estimation with nyström kernel exponential families. In International Conference on Artificial Intelligence and Statistics, pages 652-660. PMLR, 2018.
[37] Hiroki Suyari, M Tsukada, and Y Uesaka. Mathematical structure derived from tsallis entropy. In Proc. 4th Asia-Eur: Workshop Inf. Theory (AEW), pages 1-4, 2004.
[38] Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics, pages 567-574. PMLR, 2009.
[39] Lloyd N Trefethen and JAC Weideman. The exponentially convergent trapezoidal rule. siam REVIEW, 56(3):385-458, 2014.
[40] Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing JEMNL--IJCNLPP pages 4344 4353, 2019.
[41] Constantino Tsallss. Possible generalization of boltzmann- -gibss statistics. Journal of statistical physics, 52(1) 1988.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference Neural Information Processing Systems, pages 6000-6011 2017.
[43] Shaojun Wang, Dale Schuurmans, and Yunxin Zhao. The latent maximum entropy principle. ACM Transactions Knowledge Discovery from Data (TKDD), 6(2):1-42, 2012.
[44] Li Wenliang, Dougal Sutherland, Heiko Strathmann, and Arthur Gretton. Learning deep kernels for exponential family densities. In International Conference on Machine Learning, pages 6737-6746. PMLR, 2019.
[45] Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Self- attention with functional time representation learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. I'Alché-Buc. E. Fox, and R. Garnett, editors. Advances in Neu- ral Information Processing Systems, volume 32, pages 15915-1555 15925. Curran Asso- ciates, Inc., 2019. URL https://proceeding: wew.eeeeeeiiinnnnnnnniieeeeeeepp neurips cc/ pappr/2019/fiie E
.dff
[46] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. pages 2114-2124, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We did it for three of our five experiments, where the datasets were small and sequence lengths were short: the synthetic time warping experiment, the FordA experiment and the uaave experiment.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We mention the type of resources used. These relatively small scale experiments compute as environmental impacts are negligible.
don't need to discuss total
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [No]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "xJf8qouEUwG", "reviewer": "Reviewer_PeFu", "summary": "The paper generalizes the recent results (Martins et al. 2020, 2021) on continuous attention mechanisms to sparse multimodal density classes, which referred to as kernel deformed exponential families. The authors analyze theoretically three main aspects of the density families: normalization, approximation ability, and evaluation of context function. Then the authors apply the methods to real data experiments.", "questions": "1. Line 84-87: Please specify the function domain and image space of V. My understanding is that V itself is a fixed function and has nothing to do with the probability. When S is a discrete set, \\{V(t): t\\in S\\} is a sequence of values over S. You would better not use time series here to call V(t) here since traditional time series refer to a sequence of random variables across time.\n2. Line 88-90: Similar to the above comment, since the randomness comes from T variable, this is different from the traditional definition of stochastic processes.\n3. Line 93: Please give a proper reasoning of using the notation \\mathcal{M}_+^1.\n4. Line 95: Please give the definition of a proper functional.\n5. Equation (2): Please give the definition of the L2 inner product over measure Q.\n6. Line 98: Please give the definition of negative differential entropy. \n7. Equation 3: How can you reach equation 3 (you maybe can add a reference here)? What is A(f) here? Is it the normalizing constant?\n8. Equation 6: What is norm F here? Is this the Frobenius norm?\n9. Line 121-122: The sentence \"Physical time may not ...  an individualized time scale transforming physical time\" is sort of misleading. Do you want to emphasize the importance of investigating individual level of physical times where a time warping technique is called for?\n10. Definition 4.1: Can you give a more intuitive description of the time warping function? You can use some figures or examples here. Do you need t_{ij} to be an increasing sequence over j? Given the third condition for h_i (the equation under line 131), how can you make h_i strictly increasing?\n11. Lemma 4.2: Please introduce U before using in equation 7. Also in appendix, B.1, should be proof of \"Lemma\" 4.2.\n12. Section 5.2.2: Please include at least the main theorem in the main text. Otherwise, one has to treat these results in appendix less important.\n13. Section 5.3: The presentation in this section is far from clear. Please revise and give more details and explanation.\n14. Line 316: typo \"to use\". \n15. Line 77: typo \"improved\". \n16. Line 78-79: I do not get point 5 here. Also the authors write two separate sections on their contributions (at the end of Section 1 and 2). Please revise to make these more concise and clear.\n\n\n\n\n\n", "limitations": "Yes. Also see weaknesses section.", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper extends previous results on continuous attention mechanisms to kernel deformed exponential families. The authors make a great effort to apply their methods in practice. My major concerns are as follows: (1) There are many notation and terminologies used without reference or definition. (2) Given the previous results such as Martins et al. (2020,2021,2022), Farinhas etal. (2021), the theoretical novelty is limited. (3) Since numerical integration has to be implemented in practice and no reliable theoretical analysis is given, the empirical performance of the framework is questionable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IWjGzKzo8NR", "reviewer": "Reviewer_rxaS", "summary": "The authors provide sufficient conditions under which the normalising constant of a kernel exponential family exists (proposition 5.1). Furthermore, they introduce so-called deformed kernel exponential families (sec 5.2) and provide a similar normalisation theory (corollary 5.2), and show conditions under which deformed kernel exponential families are dense in a class of deformed exponential families that uses the space of continuous functions vanishing at infinity in place of the RKHS (appendix D.4). The deformed variant allows for modelling (infinite dimensional) densities with finite support (Figure 1).\n\nThese technical results, which are of broad interest, are then applied to a continuous version of kernel attention. Continuous kernel attention is framed as an expectation over a density that is the solution to a regularised strictly convex functional (2). In this case, the (deformed) kernel exponential families are used as the space from which the solution is drawn. \n\nThe authors try their new versions of kernel attention on some toy datasets, observing better performance than other methods.", "questions": "- I did not properly grasp the motivation behind studying time warping. Is because certain multimodal densities can be represented using time warping? Would it be possible to make the motivation more clear at the start of section 4?\n- Is it possible to use kernel and deformed kernel exponential families without the normalisation constant, especially in the cases where the normalisation constant does not exist? Would this ever be justified?\n- How does your notion of attention compare with the deep learning definition? Especially with respect to \"self-attention\"? Is this more close to \"self-attention\" than general attention?", "limitations": "(Not that it affects my evaluation of the paper in any way, but) IMO there is no need to include the last sentence in section 6. Hate speech is one of many potential negative societal impacts, and anyone who has read this far into the paper will already appreciate that this is a largely theoretical work without direct societal application in mind. Hate speech is a non-trivial issue. Singling out hate speech without any context earlier in the paper looks lazier than if it had not been mentioned at all, and does not respect the issue.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- I enjoyed this paper. The authors seem to have properly handled normalisation constants for kernel and deformed kernel exponential families. I like that new theory that may be of independent interest is developed. The approach is thought-out and principled, unlike some other attempts to generalise attention.\n\n\nWeaknesses:\n- I did not like the way equation (1) was presented. It was simply stated \"It is (1)\". Can you provide a reference for this definition? Or is this your definition? Also see question in Question section below.\n- In line 235, we use a finite span for a representation of the function in the RKHS. Is this invoking the RKHS representer theorem? It does not apply to the regularised problem (2) because this concerns the density itself not the f \\in RKHS. So where does this finite span representation come from? It seems like Algorithm 1 assumes the coefficients (parameters) of the representer are already known, so presumably one way to justify this is by wrapping algorithm 1 in some regularised empirical risk minimisation problem? I am happy with either a mathematically rigourous justification or a handwavy justification, as long as the handwaving is clearly admitted in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4XRUP3qcWjD", "reviewer": "Reviewer_W1Fi", "summary": "In this paper, the authors proposed a multimodal attention density based on kernel exponential families and kernel deformed exponential families for the continuous attention mechanisms. Furthermore, the authors performed the theoretical analysis about normalization, approximation capabilities and properties. In addition, the authors conducted a set of experiments to evaluate the performance of the proposed multimodal attention density, the experimental results shown that kernel continuous attention often outperforms unimodal continuous attention.", "questions": "1) what is the role of the Time warping section? Is it to prepare for either section 6.1  or to introduce Lemma 4.2?\n2) For the proof of Lemma 5.3 in the Appendix, the first equals sign may be difficult to understand for most readers. Please give the definition of α-exponential.\n3) There may be one more word in the Related work section. “They only used unimodal (deformed) exponential family densities:”\n4) For equation (6), the matrix dimensions may be incorrect. \n5) For the experimental results in Table 2 and Table 3, the standard deviation of the results is missing. \n6) Additional time and space complexity with wall-clock time reporting will be helpful.\n", "limitations": "NO", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- The concept of kernel deformed exponential families is proposed, which is a sparse multimodal density class. \n- The overall presentation of this paper is good.\n\n## Weaknesses:\n1) The motivation of this work is not well organized or elaborated for using multimodal attention density. \n2) This paper tends to be an incremental work. The authors applied kernel methods to the (deformed) exponential families to construct a multimodal attention density for continuous attention. However, the kernel exponential families deformed exponential families, and continuous attention is existing works. \n3) Experiments are not enough, why use only two performance metrics and a few datasets in the experiments section?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3ckrEK5xD6Q", "reviewer": "Reviewer_oPnf", "summary": "This paper proposes an extension of the continuous attention formalism presented by Martins et al. (2020). This extension consists in replacing the linear parameterization of the score function ($f$) in continuous attention by a kernel in an RKHS, yielding a kernel exponential family density ($\\tilde{f}$). Using q-exponentials, the authors propose using _kernel deformed exponential families_, resembling the entmax formulation of Martins et al. (2022) for $1 < \\alpha \\leq 2$, but with a special form for $f$. The main challenge in this new formulation is that computing Equation 1 with $\\tilde{f}$ is harder, and thus the authors resort to numerical integration for the forward pass and rely on automatic differentiation for the backward pass. The authors also show that numerical integration can be done efficiently with exponential convergence for kernel exponential attention. The paper presents three carefully chosen experiments to showcase multimodal continuous attention: Time Warping,  ECG heartbeat classification, and Automotive symptom detection. The results indicate that kernel multimodal continuous attention is very effective, even when compared to standard continuous softmax and sparsemax attention, and it also performs better than Gaussian mixture, which also yields multimodal densities.", "questions": "Have you encountered convergence problems with numerical integration? \nWhich hyperparameters can be tuned to get better optimizations? \nHow long does it take to run continuous kernel sparsemax compared to continuous sparsemax?\n", "limitations": "The authors have addressed the limitation and potential negative societal impact of their work adequately.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper has enough content to be appropriately divided into two main views: practical and theoretical. \n\nOn the practical side, the paper does a great job at casting deformed exponential families within the framework of continuous attention of Martins et al. (2020). The experiments were carefully chosen to illustrate the main advantages of the new formulation, mainly due to the necessity of having multimodal densities. On top of that, the authors also compared their work with related works, including the Gaussian Mixture approach by Farinhas et al. (2021). The results of the experiments are impressive as well as expected, given the importance of multimodality. The paper also presents results for text classification in Appendix G, but I believe this experiment could be in the main paper, as it was a key result reported by Martins et al. (2020). A clear downside of this work is that it requires numerical integration for the forward pass (even though the convergence is fast), which might forbid its application in specific applications. Given this discrepancy to unimodal continuous softmax/sparsemax, the paper can be improved by reporting also the runtime alongside test performance.\n\nOn the theoretical side, I believe the paper is very dense. Despite being comprehensive, the appendix is overwhelming. Some proofs could point to other works, such as D.1 -> B.3 in Martins et al. (2020). For me, it is not very clear why Proposition 5.1 and Corollary 5.2 is needed in the paper. It seems an orthogonal result to the practical side. Other than this, up to my knowledge, the theoretical aspect of the paper looks sharp.\n\nOverall, I believe the authors introduced the problem and the motivation for having multimodal continuous attention very well. The related works section is also well written. I believe that a plot from Figure 9 would further clarify the main point of the paper if it was presented in the introduction (alongside its counterpart plots in Figures 5, 6, 7, 8).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept