Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper generalizes the recent results (Martins et al. 2020, 2021) on continuous attention mechanisms to sparse multimodal density classes, which referred to as kernel deformed exponential families. The authors analyze theoretically three main aspects of the density families: normalization, approximation ability, and evaluation of context function. Then the authors apply the methods to real data experiments.
strengths_and_weaknesses: The paper extends previous results on continuous attention mechanisms to kernel deformed exponential families. The authors make a great effort to apply their methods in practice. My major concerns are as follows: (1) There are many notation and terminologies used without reference or definition. (2) Given the previous results such as Martins et al. (2020,2021,2022), Farinhas etal. (2021), the theoretical novelty is limited. (3) Since numerical integration has to be implemented in practice and no reliable theoretical analysis is given, the empirical performance of the framework is questionable.
questions: 1. Line 84-87: Please specify the function domain and image space of V. My understanding is that V itself is a fixed function and has nothing to do with the probability. When S is a discrete set, \{V(t): t\in S\} is a sequence of values over S. You would better not use time series here to call V(t) here since traditional time series refer to a sequence of random variables across time.
2. Line 88-90: Similar to the above comment, since the randomness comes from T variable, this is different from the traditional definition of stochastic processes.
3. Line 93: Please give a proper reasoning of using the notation \mathcal{M}_+^1.
4. Line 95: Please give the definition of a proper functional.
5. Equation (2): Please give the definition of the L2 inner product over measure Q.
6. Line 98: Please give the definition of negative differential entropy. 
7. Equation 3: How can you reach equation 3 (you maybe can add a reference here)? What is A(f) here? Is it the normalizing constant?
8. Equation 6: What is norm F here? Is this the Frobenius norm?
9. Line 121-122: The sentence "Physical time may not ...  an individualized time scale transforming physical time" is sort of misleading. Do you want to emphasize the importance of investigating individual level of physical times where a time warping technique is called for?
10. Definition 4.1: Can you give a more intuitive description of the time warping function? You can use some figures or examples here. Do you need t_{ij} to be an increasing sequence over j? Given the third condition for h_i (the equation under line 131), how can you make h_i strictly increasing?
11. Lemma 4.2: Please introduce U before using in equation 7. Also in appendix, B.1, should be proof of "Lemma" 4.2.
12. Section 5.2.2: Please include at least the main theorem in the main text. Otherwise, one has to treat these results in appendix less important.
13. Section 5.3: The presentation in this section is far from clear. Please revise and give more details and explanation.
14. Line 316: typo "to use". 
15. Line 77: typo "improved". 
16. Line 78-79: I do not get point 5 here. Also the authors write two separate sections on their contributions (at the end of Section 1 and 2). Please revise to make these more concise and clear.






limitations: Yes. Also see weaknesses section.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an extension of the continuous attention formalism presented by Martins et al. (2020). This extension consists in replacing the linear parameterization of the score function ($f$) in continuous attention by a kernel in an RKHS, yielding a kernel exponential family density ($\tilde{f}$). Using q-exponentials, the authors propose using _kernel deformed exponential families_, resembling the entmax formulation of Martins et al. (2022) for $1 < \alpha \leq 2$, but with a special form for $f$. The main challenge in this new formulation is that computing Equation 1 with $\tilde{f}$ is harder, and thus the authors resort to numerical integration for the forward pass and rely on automatic differentiation for the backward pass. The authors also show that numerical integration can be done efficiently with exponential convergence for kernel exponential attention. The paper presents three carefully chosen experiments to showcase multimodal continuous attention: Time Warping,  ECG heartbeat classification, and Automotive symptom detection. The results indicate that kernel multimodal continuous attention is very effective, even when compared to standard continuous softmax and sparsemax attention, and it also performs better than Gaussian mixture, which also yields multimodal densities.
strengths_and_weaknesses: This paper has enough content to be appropriately divided into two main views: practical and theoretical. 

On the practical side, the paper does a great job at casting deformed exponential families within the framework of continuous attention of Martins et al. (2020). The experiments were carefully chosen to illustrate the main advantages of the new formulation, mainly due to the necessity of having multimodal densities. On top of that, the authors also compared their work with related works, including the Gaussian Mixture approach by Farinhas et al. (2021). The results of the experiments are impressive as well as expected, given the importance of multimodality. The paper also presents results for text classification in Appendix G, but I believe this experiment could be in the main paper, as it was a key result reported by Martins et al. (2020). A clear downside of this work is that it requires numerical integration for the forward pass (even though the convergence is fast), which might forbid its application in specific applications. Given this discrepancy to unimodal continuous softmax/sparsemax, the paper can be improved by reporting also the runtime alongside test performance.

On the theoretical side, I believe the paper is very dense. Despite being comprehensive, the appendix is overwhelming. Some proofs could point to other works, such as D.1 -> B.3 in Martins et al. (2020). For me, it is not very clear why Proposition 5.1 and Corollary 5.2 is needed in the paper. It seems an orthogonal result to the practical side. Other than this, up to my knowledge, the theoretical aspect of the paper looks sharp.

Overall, I believe the authors introduced the problem and the motivation for having multimodal continuous attention very well. The related works section is also well written. I believe that a plot from Figure 9 would further clarify the main point of the paper if it was presented in the introduction (alongside its counterpart plots in Figures 5, 6, 7, 8).
questions: Have you encountered convergence problems with numerical integration? 
Which hyperparameters can be tuned to get better optimizations? 
How long does it take to run continuous kernel sparsemax compared to continuous sparsemax?

limitations: The authors have addressed the limitation and potential negative societal impact of their work adequately.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors proposed a multimodal attention density based on kernel exponential families and kernel deformed exponential families for the continuous attention mechanisms. Furthermore, the authors performed the theoretical analysis about normalization, approximation capabilities and properties. In addition, the authors conducted a set of experiments to evaluate the performance of the proposed multimodal attention density, the experimental results shown that kernel continuous attention often outperforms unimodal continuous attention.
strengths_and_weaknesses: ## Strengths:
- The concept of kernel deformed exponential families is proposed, which is a sparse multimodal density class. 
- The overall presentation of this paper is good.

## Weaknesses:
1) The motivation of this work is not well organized or elaborated for using multimodal attention density. 
2) This paper tends to be an incremental work. The authors applied kernel methods to the (deformed) exponential families to construct a multimodal attention density for continuous attention. However, the kernel exponential families deformed exponential families, and continuous attention is existing works. 
3) Experiments are not enough, why use only two performance metrics and a few datasets in the experiments section?

questions: 1) what is the role of the Time warping section? Is it to prepare for either section 6.1  or to introduce Lemma 4.2?
2) For the proof of Lemma 5.3 in the Appendix, the first equals sign may be difficult to understand for most readers. Please give the definition of α-exponential.
3) There may be one more word in the Related work section. “They only used unimodal (deformed) exponential family densities:”
4) For equation (6), the matrix dimensions may be incorrect. 
5) For the experimental results in Table 2 and Table 3, the standard deviation of the results is missing. 
6) Additional time and space complexity with wall-clock time reporting will be helpful.

limitations: NO
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors provide sufficient conditions under which the normalising constant of a kernel exponential family exists (proposition 5.1). Furthermore, they introduce so-called deformed kernel exponential families (sec 5.2) and provide a similar normalisation theory (corollary 5.2), and show conditions under which deformed kernel exponential families are dense in a class of deformed exponential families that uses the space of continuous functions vanishing at infinity in place of the RKHS (appendix D.4). The deformed variant allows for modelling (infinite dimensional) densities with finite support (Figure 1).

These technical results, which are of broad interest, are then applied to a continuous version of kernel attention. Continuous kernel attention is framed as an expectation over a density that is the solution to a regularised strictly convex functional (2). In this case, the (deformed) kernel exponential families are used as the space from which the solution is drawn. 

The authors try their new versions of kernel attention on some toy datasets, observing better performance than other methods.
strengths_and_weaknesses: Strengths:
- I enjoyed this paper. The authors seem to have properly handled normalisation constants for kernel and deformed kernel exponential families. I like that new theory that may be of independent interest is developed. The approach is thought-out and principled, unlike some other attempts to generalise attention.


Weaknesses:
- I did not like the way equation (1) was presented. It was simply stated "It is (1)". Can you provide a reference for this definition? Or is this your definition? Also see question in Question section below.
- In line 235, we use a finite span for a representation of the function in the RKHS. Is this invoking the RKHS representer theorem? It does not apply to the regularised problem (2) because this concerns the density itself not the f \in RKHS. So where does this finite span representation come from? It seems like Algorithm 1 assumes the coefficients (parameters) of the representer are already known, so presumably one way to justify this is by wrapping algorithm 1 in some regularised empirical risk minimisation problem? I am happy with either a mathematically rigourous justification or a handwavy justification, as long as the handwaving is clearly admitted in the paper.
questions: - I did not properly grasp the motivation behind studying time warping. Is because certain multimodal densities can be represented using time warping? Would it be possible to make the motivation more clear at the start of section 4?
- Is it possible to use kernel and deformed kernel exponential families without the normalisation constant, especially in the cases where the normalisation constant does not exist? Would this ever be justified?
- How does your notion of attention compare with the deep learning definition? Especially with respect to "self-attention"? Is this more close to "self-attention" than general attention?
limitations: (Not that it affects my evaluation of the paper in any way, but) IMO there is no need to include the last sentence in section 6. Hate speech is one of many potential negative societal impacts, and anyone who has read this far into the paper will already appreciate that this is a largely theoretical work without direct societal application in mind. Hate speech is a non-trivial issue. Singling out hate speech without any context earlier in the paper looks lazier than if it had not been mentioned at all, and does not respect the issue.
review_title: null

Meta-review: metareview: This is solid contribution overall. The paper is well written and the notation easy-to-follow. The authors spent a lot of effort to address the clarity issues by reviewer PeFu. Requiring numerical integration is a limitation but it is clearly acknowledged in the paper.

We recommend acceptance.

Minor additional remarks:

- The term "value function" was a bit confusing because it has a different meaning in game theory or optimal control.
- Section 4 on time warping does not cite any work. It would be great to connect it better with the existing literature.
confidence: Certain
Decision: Accept