Title: FAME: Adaptive Functional Attention with Expert Routing for Function-on-Function Regression
Venue: NeurIPS 2025
Accepted: yes
Keywords: Functional Attention; Function-on-Function Regression;Bidirectional NCDE;Mixture-of-Experts
Abstract: Functional data play a pivotal role across science and engineering, yet their infinite-dimensional nature makes representation learning challenging. Conventional statistical models depend on pre-chosen basis expansions or kernels, limiting the flexibility of data-driven discovery, while many deep-learning pipelines treat functions as fixed-grid vectors, ignoring inherent continuity. In this paper, we introduce Functional Attention with a Mixture-of-Experts (FAME), an end-to-end, fully data-driven framework for function-on-function regression. FAME forms continuous attention by coupling a bidirectional neural controlled differential equation with MoE-driven vector fields to capture intra-functional continuity, and further fuses change to inter-functional dependencies via multi-head cross attention. Extensive experiments on synthetic and real-world functional regression benchmarks show that FAME achieves state-of-the-art accuracy and strong robustness to arbitrarily sampled discrete observations of functions.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents FAME, a deep learning framework for function-on-function regression (FoFR). The model is designed to work with irregularly sampled functional data and produces continuous functional outputs. It combines several ideas: (1) bidirectional neural controlled differential equations (NCDEs) for capturing intra-functional structure, (2) a mixture-of-experts (MoE) router for modeling heterogeneity across functions, and (3) cross attention for learning dependencies across input channels. The authors claim that FAME provides theoretical guarantees such as Lipschitz stability, sampling invariance, and universal approximation. Experimental results on synthetic and real-world datasets show that FAME outperforms various statistical and neural baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The architecture is thoughtfully designed and well-motivated for the challenges of FoFR.\n- The theoretical analysis includes Lipschitz bounds and approximation results, which help support the method's generality and stability.\n- Extensive experiments on both synthetic and real-world datasets show FAME achieves strong performance, especially in irregular or noisy settings.\n\nWeaknesses:\n- The paper doesn't report any computational metrics. Given that the model relies on NCDE solvers and attention mechanisms, it's important to understand how it compares to lighter models in terms of runtime or memory use.\n- Appendix A contains the theoretical proofs, but it's hard to trace which result in the appendix corresponds to which theorem in the main text. The connection between Theorem 3 and the appendix, for instance, is unclear.\n- No ablation studies are provided. It would be helpful to know how much each component (e.g., MoE, bidirectionality) contributes.\n\nQUESTIONS:\n- Could you provide runtime and memory usage for FAME? Since the model uses NCDEs and attention, it may be more computationally demanding than classical methods. It would be helpful to understand how the training cost compares to baselines like FPCA, kernel methods, or functional neural networks.\n- The proofs in Appendix A don't clearly map to the theorems stated in the main text. For example, Theorem 3 in the main paper doesn't appear to have an explicit label in the appendix. It would improve clarity if each theoretical result in the main paper pointed directly to its proof in the appendix using consistent numbering.\n- Have you tested the model's behavior on longer or more densely sampled functional data? Since NCDEs can be computationally intensive, some indication of the method's scalability would be useful.\n- You mention running five random splits for experiments, but the results are presented without error bars. Including standard deviations or confidence intervals would help readers assess the statistical significance of the improvements.\n\nLIMITATIONS:\nThe authors mention that FAME may not be optimal in low-complexity settings where simpler models suffice. However, the paper does not discuss the method's computational cost or efficiency relative to baselines, which is important for practical deployment. This should be acknowledged more explicitly.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces FAME, a novel framework for function-on-function regression (FoFR) that combines bidirectional neural controlled differential equations (NCDEs) with a mixture-of-experts (MoE) architecture and continuous attention mechanisms. The proposed model aims to address challenges such as intra-functional continuity, inter-functional interactions, and feature heterogeneity in functional data. The authors demonstrate FAME's superiority over existing methods through extensive experiments on synthetic and real-world datasets, showcasing its robustness to irregular sampling and feature variability.\n\nSTRENGTHS AND WEAKNESSES:\nOriginality: The contribution of this paper lies in introducing MoE and attention mechanisms into NCDEs. This combination is possibly a novel attempt, but its components are all existing techniques.\n\nClarity: The paper is clearly written. The main pipeline is easy to understand. The formulas used seem correct (I didn’t check the lemmas, Theorems, etc. in the paper). \n\nQuality: The paper supports its claims through theoretical analysis and experiments. The combination of NCDEs, MoE, and attention mechanisms is well-justified and effectively addresses the challenges of FoFR.\n\nSignificance: This study solves an important problem in functional data analysis and provides a feasible solution that is superior to existing methods and has potential real-world application value.\n\n## Weaknesses\n\n1. The proposed FAME appears to be a simple combination of NCDEs, MoE and attention mechanisms without improvements in details.\n\n2. The paper lacks in-depth analysis of the introduced MoE and attention mechanisms. How was the result of expert selection in MoE? Was the expert selection sparse, or did it have some regular patterns? Also, what did the attention maps reveal? Which factors (e.g., function similarity, temporal dependency) would lead to high attention scores between functions?\n\n3. The paper did not provide a detailed analysis of the computational cost of FAME, especially compared to simpler baselines. The overhead of MoE routing and attention mechanisms could be prohibitive for large-scale applications.\n\nQUESTIONS:\n1. Provide in-depth analysis of the introduced MoE and attention mechanisms.\n2. Analysis on the computational cost of FAME.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces FAME, a continuous‑time architecture for function‑on‑function regression that couples bidirectional Neural Controlled Differential‑Equation encoders with a per‑function mixture‑of‑experts router and a cross‑attention fusion block. This design preserves the ability to query predictions at arbitrary points on irregular input and output grids while allowing each input function to adapt its vector field to idiosyncratic scale, frequency, and noise characteristics. A theoretical analysis establishes existence, uniqueness and global Lipschitz stability for every differential block, yields an overall Lipschitz constant for the end‑to‑end operator, and converts that constant into a Rademacher‑complexity generalisation bound. Experiments on seven synthetic regimes built from Gaussian‑process trajectories and on three real‑world datasets show lower test‑set MSE than classical FDA baselines, Gaussian processes, and recent deep‑learning competitors.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n+ The paper’s main conceptual novelty lies in replacing discrete self‑attention with an NCDE‑based continuous attention mechanism, thereby treating both inputs and outputs as genuine functions rather than large vectors. The mixture‑of‑experts router is carefully scoped, one set of weights per function instead of per time step, so it captures inter‑function heterogeneity without incurring prohibitive computational cost. \n\n+ The theoretical component is complete for work: all assumptions are stated explicitly, constants are propagated through the proofs and a non‑trivial Rademacher bound is obtained. On the empirical side, the study is broad, covering diverse sampling patterns, noise regimes, and application domains; the model wins almost every column of the main tables and converges reliably in the training curves shown in the appendix. \n\n+ The paper is well organised; implementation details, hyperparameters, and hardware are documented in the supplementary material, and the authors commit to releasing code and checkpoints upon acceptance.\n\n\nWeaknesses\n\n- The comparison set mainly covers classical FDA techniques. Apart from a functional‐neural-network baseline, every competitor is either a fixed-basis regression, FPCA, a Gaussian-process regressor or an operator-valued kernel; no recent continuous-time deep architectures such as transformer-style sequence models or neural operators are evaluated, even though the related-work section cites them. Because these models are designed to exploit long-range dependencies and irregular grids, omitting them leaves open whether FAME’s margin would persist against the current state of the art.\n\n- For synthetic data, the authors average over only five random train/test splits and quote the mean MSE without standard deviations; for all real-world tasks, the results come from a single split and single seed, with no confidence intervals or significance tests. Without variability measures, it is hard to judge whether the observed gains are robust.\n\n- The supplementary material removes the mixture-of-experts router in one heterogeneity test, but there is no experiment that isolates the benefit of bidirectionality, the cross-attention module, or the continuous-attention block itself across the full benchmark suite. As a result the individual contribution of each architectural component remains unclear.\n\n- Although the theory rests only on deterministic Lipschitz and boundedness conditions, these are assumed to hold globally for every expert vector field and decoder. In real high-dimensional applications global Lipschitz constants can be large or undefined; the paper does not explore how sensitive the Lipschitz stability or the Rademacher bound is to mild violations of these assumptions.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes a method for solving function-on-function regression tasks. The method entitled FAME does this through learning a mixture of experts and has full attention, employing bidirectional NCDE and continuous attention. This approach ensures continuity, sampling invariance, and Lipschitz stability. This method is very flexible, working for input and output data with potentially distinct irregular grids. This paper has very strong quantitative performance across many datasets including both synthetic and real world data.

STRENGTHS AND WEAKNESSES:
(+) This paper is really clear and well written.

(+) There is substantial evaluation across various challenging input and output sampling schemes such as irregular grid, different sampled points, and noisy observations. There is also strong quantitative performance on real-world datasets against various baselines.

(-) Limited discussion on how hyperparameters (e.g. $K$ and $d$) are chosen.

(-) Additional ablation studies would be helpful to understand which components of the method are the most impactful. Why is the cross attention necessary when there is continuous attention?

QUESTIONS:
- How do you decide on the number of experts $K$? Is this assigned a priori depending on how complex you think the data will be? How does changing $K$ affect your results?
- What are the hyperparameters (e.g. $K$, $d$) used for different real data tests and how did they get selected? 
- Why is the cross attention necessary when there is continuous attention?

LIMITATIONS:
A limitation that the authors include is that simpler methods may be preferred when the underlying operator varies little or is simpler.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02