Title: Learning to Route: Per-Sample Adaptive Routing for Multimodal Multitask Prediction
Venue: NeurIPS 2025
Accepted: yes
Keywords: multimodal learning

multitask learning

adaptive routing

per-sample adaptation

heterogeneous data

clinical applications

routing networks
Abstract: We propose a unified framework for adaptive routing in multitask, multimodal prediction settings where data heterogeneity and task interactions vary across samples. We introduce a routing-based architecture that dynamically selects modality processing pathways and task-sharing strategies on a per-sample basis. Our model defines multiple modality paths, including raw and fused representations of text and numeric features, and learns to route each input through the most informative modality-task expert combination. Task-specific predictions are produced by shared or independent heads depending on the routing decision, and the entire system is trained end-to-end. We evaluate the model on both synthetic data and real-world psychotherapy notes, predicting depression and anxiety outcomes. Our experiments show that our method consistently outperforms fixed multitask or single-task baselines, and that the learned routing policy provides interpretable insights into modality relevance and task structure. This addresses critical challenges in personalized healthcare by providing per-subject adaptive information processing that accounts for data and task correlation heterogeneity.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a unified framework for per-sample adaptive routing in multimodal, multitask prediction, focusing on mental health applications. It dynamically selects optimal data pathways (structured, unstructured, or fused) and task-sharing strategies (single-task or multitask) for each input using a probabilistic gating mechanism. Evaluated on synthetic data and real psychotherapy records, the model outperforms fixed baselines in predicting depression and anxiety outcomes.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. **Flexible and Unified Framework**: The paper introduces a modular architecture capable of adaptively routing inputs across multiple modality combinations and task-sharing strategies. This per-sample adaptivity is well-motivated in clinical prediction tasks with heterogeneous data.\n\n2. **Interpretable Routing Decisions**: The probabilistic routing mechanism yields interpretable outputs that align with clinical expectations, offering potential benefits for real-world deployment in healthcare settings.\n\n---\n\n**Weaknesses:**\n\n1. **Limited Real-World Validation**: A substantial portion of the experiments relies on synthetic data, which may not accurately reflect real-world complexities or generalize well to clinical applications. The use of hybrid synthetic-real data further undermines confidence in the method's effectiveness in practical settings.\n\n2. **Lack of Ablation Studies**: The paper does not provide sufficient ablation experiment details to support the contribution of each component (e.g., routing, heteroscedastic loss, modality transformations). This limits the reader’s ability to assess what aspects of the model drive performance improvements.\n\n3. **Weak Baseline Comparisons**: The baselines are limited and insufficiently explored. The claimed gains over simple fixed-path models may result more from soft averaging than meaningful architectural innovation.\n\n4. **Low Originality**: The core idea—softmax-based selection over modality-task paths—is a straightforward extension of the feature selection and shared decoder technique, and I don't see why it is called MoE. The conceptual novelty is modest, and the methodological contribution may not be significant enough for a top-tier venue like NeurIPS.\n\nQUESTIONS:\n**Questions and Suggestions**\n\n1. **Ablation Study Design**\n   Could the authors clarify how the ablation studies were conducted, if any? Specifically:\n\n   * Were parameter counts controlled across ablated variants?\n   * Were all models (including fixed-path baselines) trained with the same set of hyperparameters?\n   * If not, how do the authors ensure that performance improvements are due to the adaptive routing mechanism rather than model capacity or training discrepancies?\n     A detailed, fair ablation setup is essential to isolate the contribution of each component. A clear answer here would improve confidence in the reported gains.\n\n2. **Clarity on MoE Design and Expert Modules**\n   The paper claims to adopt a Mixture-of-Experts (MoE) framework, but the actual architectural design seems limited to gating over modality/task paths—without clearly defined expert subnetworks. Could the authors:\n\n   * Clarify what constitutes an \"expert\" in their setup?\n   * Specify whether each expert has independent parameters, or whether the same backbone is reused with only gating on inputs/outputs?\n     As it stands, the model appears closer to a soft ensemble over different input transformations than a true MoE. A detailed architectural diagram or parameter-sharing explanation would help clarify this point. If experts are not truly independent, the originality and significance of the method may need to be reconsidered.\n3. **Baseline Selection and Use of Recent Models**\n   Have the authors evaluated their approach against stronger or more recent baselines, such as:\n\n   * Standard BERT-based encoders with task-specific heads?\n   * Multimodal fusion models using pretrained language models with structured data adapters (e.g., GatorTron-style or Med-BERT variants)?\n     Given the prevalence of BERT-like models in clinical NLP, it is important to understand how the proposed method compares to these widely used, stronger baselines. Including such comparisons would better contextualize the reported performance gains and strengthen the case for the proposed approach.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a unified framework for per-sample adaptive routing in multimodal, multitask prediction, motivated by applications in psychotherapy where both structured assessments (e.g. questionnaires) and unstructured clinician notes coexist. The key idea is to dynamically route each input through different modality transformations (e.g. text-only, numerical-only, or fused representations) and task architectures (single-task vs. multitask), depending on input-specific signals. The model employs a hierarchical mixture-of-experts architecture with learnable routing networks that select both the optimal modality pathways and the degree of task sharing for each sample.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- **Novelty in Joint Adaptive Routing** The paper introduces a unified adaptive routing framework that jointly routes over both modalities and task structures (STL vs. MTL) on a per-sample basis. This is a meaningful extension beyond typical MoE architectures, which often focus solely on routing over experts for a fixed modality or task.\n- **Bidirectional Modality Transformations** The bidirectional transformations between structured and unstructured data (numerical → text and text → numerical) are a creative and flexible mechanism. They enable four distinct fusion paradigms (T1, N1, T2, N2) that adapt to missing or weak modalities, addressing a genuine practical challenge in clinical data.\n- **Strong Clinical Motivation and Application** The motivation is rooted in a real-world clinical problem and the paper is highly relevant for personalized medicine\n- **Uncertainty-Aware Modeling** The paper combines adaptive routing with a heteroscedastic loss, modeling prediction uncertainty.\n\n**Weaknesses**\n- **Reliance on Synthetic and Hybrid Data** The real-world experiments partially rely on a hybrid dataset that augments genuine psychotherapy records with synthetic encounters. This introduces potential domain shift, making it unclear how the model would perform on entirely real, diverse clinical datasets.\n- **Limited Task Diversity** The experimental validation focuses on only two clinical outcomes (PHQ-9, GAD-7). While well-motivated, this leaves open the question of whether the method generalizes to more diverse mental health outcomes and other clinical domains\n- **Computational Scalability Not Analyzed** The model considers eight expert paths (4 modality paths × 2 task configurations). However, there's no explicit analysis of computational cost. How does routing scale with increasing numbers of modalities, tasks, or larger datasets? What is the inference-time cost for large-scale deployment?\n- **Potential Overhead from Soft Routing** The soft routing approach computes a full mixture over all expert paths, requiring all experts to run per sample. The paper does not quantify the runtime penalty of soft vs. hard routing, nor how this affects latency in practice.\n\nQUESTIONS:\n- Soft routing requires computing outputs for all 8 expert paths per sample. Did you measure the inference-time computational cost of your method compared to fixed-path baselines? How significant is the overhead, especially for larger datasets?\n\n- You mention using entropy regularization to prevent expert collapse. Could you quantify how much this regularization affects routing diversity in practice? Without it, did you observe any experts dominating the routing probability mass?\n\n- Your results show that the textualized numeric + text path (T2) outperforms numerical + embedded text (N2). Could you elaborate on why textualization yields better performance? Do you believe this is due to language models capturing richer context, or could it reflect overfitting to the textual modality?\n\n- Do you foresee risks or limitations in transforming structured numeric data into textual form? For example, might subtle quantitative nuances be lost or misrepresented during textualization, especially for clinical scales?\n\n- Beyond aggregate routing probabilities, have you explored visualizing routing decisions at the individual patient level? Could such visualizations aid clinicians in understanding why the model made certain predictions?\n\nLIMITATIONS:\nLimitations have been discussed in section 5.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel framework for adaptive routing in multimodal multitask prediction, specifically designed to handle data heterogeneity and varying task interactions in complex real-world settings like psychotherapy. The proposed architecture dynamically selects modality processing pathways and task-sharing strategies on a per-sample basis, enabling the model to adapt to individual input characteristics and task correlations. This is achieved through a routing mechanism that chooses the most informative modality paths and expert combinations, producing task-specific predictions through shared or independent heads. The framework is evaluated on synthetic data and real-world psychotherapy notes, demonstrating superior performance over fixed multitask and single-task baselines. This approach has significant implications for personalized healthcare, potentially improving mental health outcomes and treatment precision.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\n\nStrengths:\nThe paper presents a robust and well-structured framework, with comprehensive experiments that validate the model's effectiveness in both synthetic and real-world settings.\nThe use of probabilistic routing to manage sample-specific heterogeneity is a strong methodological contribution, allowing for nuanced handling of data variability and task interdependencies.\nThe results demonstrate clear performance improvements over baseline models, showcasing the framework's practical benefits in terms of accuracy and interpretability.\n\nWeaknesses:\nWhile the framework is well-designed, the reliance on synthetic data for initial validation might not fully capture the complexities encountered in entirely real-world datasets.\nThe implementation details, particularly regarding the neural network architectures used for routing and expert selection, could be more thoroughly explained to enhance reproducibility.\n\nClarity:\nStrengths:\nThe paper is generally well-organized, providing clear explanations of the adaptive routing framework and the underlying concepts.\nVisual aids such as figures and tables effectively illustrate the model architecture, routing behavior, and experimental results, aiding reader comprehension.\n\nWeaknesses:\nSome technical aspects, such as the specific algorithms used for training the routing mechanism and expert modules, could benefit from additional detail to assist readers in understanding the implementation.\nThe integration of related work could be more cohesive, providing clearer context for how the proposed method differs from or improves upon existing approaches.\n\nSignificance:\n\nStrengths:\nThe paper addresses a critical challenge in multimodal multitask learning, offering a novel solution that could significantly enhance predictive performance in settings with heterogeneous data and complex task relationships.\nThe framework's potential applications in personalized healthcare and mental health prediction are particularly impactful, promising improvements in clinical decision-making and treatment outcomes.\n\nWeaknesses:\nThe broader applicability of the framework in other domains beyond mental health is not fully explored, which could limit its perceived significance in a wider context.\nThe potential impact on clinical workflows and how the model's interpretability could be leveraged in practice is mentioned but not deeply analyzed.\n\nOriginality:\n\nStrengths:\nThe introduction of a probabilistic routing mechanism that dynamically selects modality and task configurations on a per-sample basis is a novel contribution, setting this framework apart from existing multimodal and multitask learning approaches.\nThe integration of adaptive routing with heteroscedastic loss further enhances the model's originality, addressing both data variability and uncertainty in predictions.\n\nWeaknesses:\nWhile the approach is innovative, it builds on established concepts like mixture-of-experts and adaptive computation, which might moderate perceptions of originality.\nThe focus on specific clinical applications, while significant, may narrow the scope of originality if not extended to other potential domains.\n\nQUESTIONS:\nReal-World Applicability:\nThe reliance on synthetic data for initial validation is a notable aspect of the framework. How do you envision the model performing in entirely real-world scenarios where data distributions may be more complex and nuanced? It would be useful to discuss potential strategies for adapting the framework to such environments and mitigating any synthetic bias.\nInterpretability and Clinical Workflow Integration:\nWhile the paper mentions the interpretability of the routing decisions, could you elaborate on how these insights could be practically integrated into clinical workflows? Discussing specific use cases or examples where the model's decisions could enhance clinical decision-making would strengthen its applicability.\nBroader Applicability:\nThe framework is primarily validated in the context of mental health prediction. How do you anticipate it performing in other domains with multimodal and multitask challenges? Exploring potential extensions or adaptations for broader applications would enhance the paper's significance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a unified framework for adaptive routing in multimodal multitask prediction, addressing data heterogeneity and task interactions across samples. The model dynamically selects modality processing pathways (e.g., text-only, numerical-only, fused representations) and task-sharing strategies (single-task learning [STL] or multitask learning [MTL]) on a per-sample basis. Experiments on synthetic and real-world psychotherapy data show improved prediction accuracy over fixed baselines, with interpretable routing policies that align with clinical intuition.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1.  Novel Framework: The integration of modality and task routing in a single framework addresses critical limitations of fixed multimodal multitask models, especially in handling heterogeneous data.\n\n2. Clinical Relevance: The approach is tailored to psychotherapy data, where structured and unstructured inputs coexist with missing values and correlated outcomes. The real-world experiments on mental health prediction show practical utility.\n\n3. Interpretability: The learned routing policies provide insights into modality-task relationships, which is valuable for clinical applications requiring transparency.\n\n**Weaknesses:**\n\n1. Real-World Data Limitations: The dataset involves augmented records with synthetic encounters, which may introduce distribution shifts. Validation on untouched clinical data is needed.\n\n2. Computational Overhead: The mixture-of-experts approach with soft routing could incur higher computational costs, though the paper does not explicitly analyze efficiency.\n\n3. Generalization to Other Domains: The framework is evaluated primarily in clinical settings; its applicability to other multimodal domains remains untested.\n\nQUESTIONS:\n1. Data Augmentation Impact: The real-world experiment uses a hybrid corpus with synthetic encounters. Please clarify how the model performs on purely genuine clinical data.\n\n2. Computational Efficiency: Could the authors provide details on the computational overhead of the routing mechanism compared to fixed baselines.\n\n3. Modality Transformation: The bidirectional transformations (numerical-to-text and text-to-numerical) are central to the framework. Please provide ablation studies to quantify the impact of each transformation path on prediction accuracy and routing decisions.\n\n4. Scalability to More Tasks/Modalities: How does the framework scale when handling more than two tasks or additional modalities.\n\n**Update**\nThank the authors for the rebuttal. I have also review the comments from other reviewers. \n\nFrom my perspective, the novelty of method in this work is limited. \nThe proposed framework is a direct application of the Multi-gate Mixture-of-Experts (MMoE) architecture. The paper fails to articulate a clear distinction from MMoE or demonstrate why this specific formulation is superior.\n\nMoreover, the experiments on multimodal data is insufficient. Therefore I decide to maintain my original score.\n\nLIMITATIONS:\nThe paper discusses limitations (e.g., synthetic data biases, hybrid corpus) and potential negative impacts (e.g., misinterpretation of results in clinical settings). The authors propose future work to validate on real-only datasets and mitigate bias, which is commendable.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}