Learning to Route: Per-Sample Adaptive Routing for Multimodal Multitask Prediction
Authors: Marzieh Ajirak
Oded Bein
Ellen Rose Bowen
Dora Kanellopoulos
Avital Falk
FAITH M. GUNNING
Nili Solomonov
Logan Grosenick
Venue: NeurIPS 2025 poster ()
Keywords: multimodal learning
multitask learning
adaptive routing
per-sample adaptation
heterogeneous data
clinical applications
routing networks
Primary area: deep_learning
TL;DR: We propose a model that learns per-sample routing for multimodal multitask prediction, improving accuracy and interpretability on heterogeneous data.

Abstract: We propose a unified framework for adaptive routing in multitask, multimodal prediction settings where data heterogeneity and task interactions vary across samples. We introduce a routing-based architecture that dynamically selects modality processing pathways and task-sharing strategies on a per-sample basis. Our model defines multiple modality paths, including raw and fused representations of text and numeric features, and learns to route each input through the most informative modality-task expert combination. Task-specific predictions are produced by shared or independent heads depending on the routing decision, and the entire system is trained end-to-end. We evaluate the model on both synthetic data and real-world psychotherapy notes, predicting depression and anxiety outcomes. Our experiments show that our method consistently outperforms fixed multitask or single-task baselines, and that the learned routing policy provides interpretable insights into modality relevance and task structure. This addresses critical challenges in personalized healthcare by providing per-subject adaptive information processing that accounts for data and task correlation heterogeneity.

Learning to Route: Per-Sample Adaptive Routing for Multimodal Multitask Prediction
Marzieh Ajirak Oded Bein' Ellen Rose Bowen'
Dora Kanellopoulos'
Avital Falk'
Faith Gunning'
Nili Solomonov Logan Grosenick 1,27
Weill Cornell Medicine, Cornell University, NY, USA Feil Family Brain Mind Research Institute, Cornell University, NY, USA
Abstract
We propose unified framework for adaptive routing in multitask, multimodal prediction settings where data heterogeneity and task interactions vary across sam- ples. We introduce routing-basse based architecture that dynamically selects modality processing pathways and task-sharing strategies on per-sample basis. Our model defines multiple modality paths, including raw and fused representations of text and numeric features, and learns to route each input through the most informa- tive modality expert combination. Task-specific predictions are produced by shared or independent heads depending on the routing decision, and the entire system is trained end-to-end. We evaluate the model on both synthetic data and real-world psychotherapy notes, predicting depression and anxiety outcomes. Our experiments show that our method consistently outperforms fixed multitask or single-tasi baselines, and that the learned routing policy provides interpretable insights into modality relevance and task structure. This addresses critical chal- lenges in personalized healthcare by providing per-subject adaptive information processing that accounts for data and task correlation heterogeneity.
Introduction
Modern predictive models increasingly operate in settings with multiple heterogeneous input modali- ties and correlated outputs. In domains such clinical informatics and behavioral health (Preis et al.) 2022), data arrive in diverse formats that include structured numerical features (clinical scales, sensor measurements) and unstructured text (clinician notes, patient narratives) (Baltrušaitis al. 2018 Rajkomar el al., 2018). These modalities differ in structure, coverage, semantic density, and sample- level informativeness. On the other hand, multiple predictive targets (for example, depression and anxiety scores) often correlate but do not overlap entirely. This combination motivates multimodal multitask learning models that integrate heterogeneous inputs while modeling structured relationships across tasks.
Multimodal learning aims learn joint representations from diverse data sources. However, most existing approaches assume fixed fusion strategies and complete modality availability (Ruder) [0017] Liu et al. 2022). Similarly, multitask learning (MTL) typically relies on globally shared architec- tures, applying the same parameter sharing scheme to all inputs. However, these assumptions are often violated in real-world settings. Modality quality and informativeness can vary substantially across samples, and task relationships may differ depending on latent factors such as individual behavior, context, or data completeness. Ignoring these forms of heterogeneity leads to suboptimal representations and reduced predictive performance and generalization.
*Corresponding authors: log4002 @ med.cornllleduu I Joint senior authors
Marzich Ajirak, maa4083 @ mod.colmlll.cdu;
Logan Grosenick,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To address these limitations, we propose a unified framework that performs adaptive expert routing over both modality and task configurations (Shazeer et al. 2017 Ma et al. 2018 Rosenbaum et al., 2017). Our model defines a set of modality transformation paths, including unimodal and fused representations, and set of prediction heads corresponding to single-task and multitask supervision structures. A learnable routing mechanism selects a personalized expert pathway for each input by modeling a probabilistic mixture over modality-task combinations, capturing both input-dependent modality preferences and task-level interaction structure. This approach generalizes conditional computation and mixture-of-experts (MoE) frameworks (Liu et al. 2024a) to settings where latent structure exists across inputs, supervision targets, and representational hierarchies.
The routing policy is parameterized by neural networks and trained jointly with all expert modules using backpropagation. To promote policy diversity and mitigate expert collapse (where subset of experts get nearly all the traffic), we incorporate an entropy regularization term (Fedus et al.) 2022). This design enables sample-specific selection of both data representations and task decoders, effectively adapting the computation graph to underlying data geometry and supervision structure.
We validate the proposed framework through series of experiments on synthetic and real-world data. In synthetic data with controlled variation in modality relevance and task correlation, our model outperforms fixed multitask and single-task baselines while recovering interpretable routing policies. In real-world psychotherapy data with structured assessments and unstructured clinician notes (Benton et al. 2017] Niu et al. (004), the model improves prediction of anxiety and depression outcomes and reveals routing decistons that align with intuitive moddllly interactions. These results demonstrate that adaptive routing over representation and supervision structure is powerful mechanism for modeling heterogeneous, multimodal prediction tasks.
In summary, our major contributions are: (1) we develop modular architecture supporting multiple modality transformation paths and adaptive task-sharing schemes, (2) we design a probabilistic routing mechanism that dynamically selects, for each input, both optimal modality pathways and task configurations based on input and output structure, and (3) we demonstrate significant improvements in both prediction accuracy and interpretability across synthetic and real-world clinical datasets. Our approach consistently outperforms standard multitask and single-task baselines, with immediate po- tential applications for enhancing decision support in mental healthcare and broader medical contexts. The framework's 's ability to adapt to heterogeneous data while modeling structured relationships across tasks makes it particularly valuable for real-world clinical environments where data heterogeneity and quality vary (across patients, clinicians, sites, etc.) and outcome measures are often interdependent (e.g., multiple clinical scales or physiological measurements), Open source code is available at: https://ggt thub. www hrmm/rrseenick- Lab-niiiik----------ee
2 Related work
2.1 Multimodal learning in clinical and mental health contexts
growing body of research investigates multimodal learning with structured clinical data (c.g.. electronic health records, standardized assessments) and unstructured text (c.g., clinician notes) to improve outcome prediction. In general medical AI, combining tabular EHR features with narrative notes has led to measurable gains in predictive performance (Lyu et al.) (022). In the mental health domain, fusion of structured and unstructured data has yielded similar benefits. For instance, Garriga et al. (2023) predicted 28-day psychiatric crisis risk using both structured EHR variables and clinical note text, reporting that models leveraging both modalities outperformed unimodal baselines. Other studies have found that incorporating text embeddings derived from models like BERT into structured-input pipelines improves accuracy across various clinical tasks Ye et al. 2024).
However. the gains from multimodal fusion are not consistent across settings. For example, Kotula al. (2023) found that augmenting vital signs and lab values with concept-extracted notes led to only marginal improvements for ICU deterioration prediction. These mixed results suggest that the informativeness of unstructured text and the chosen fusion strategy can critically impact performance. Our approach addresses this limitation by supporting adaptive fusion (Sahu and Vechtomova] [02T]] Xue and Marculescu, 2023 Ajirak et al. 2023b). Instead of using a fixed integration schema, the model determines how to combine structured and unstructured inputs separately for each sample. This enables more flexible use of the available modalities, depending on their informativeness for the individual case.
2

---

2.2 Multitask learning for mental health prediction
Multitask learning (MTL.) (Kendall et al. 2018) offers a principled approach for modeling multiple correlated clinical outcomes. Conditions such as major depressive disorder and generalized anxiety frequently co-occur and exhibit overlapping symptom profiles, yet clinical models often treat them as independent targets or apply joint statistical models that fail to capture their structured dependencies. MTL enables parameter sharing across tasks, which supports the learning of shared representations that can improve generalization and predictive performance. Prior work has demonstrated the effectiveness of this approach in mental health settings. For example, [Buddhitha and Inkpen (2023) constructed shared encoder with both hard and soft parameter sharing to jointly model mental illness and suicide ideation risk, outperforming task-specific baselines. In another study, [Saylam and Incel (224)) showed that jointly predicting depression, anxiety, and stress levels led to performance gains for depression and stress compared to single-task models. These findings show that MTL architectures with shared encoders and task-specific decoders can leverage cross-task structure while preserving task specialization.
resktdeeatedness plays a central role in the effectiveness of multitask learning. When tasks are closely correlated or reflect underlying causal relationships, multitask models often achieve positive transfer (Standley et al.| (2020). In contrast, unrelated tasks can lead to negative transfer and representation interterence (Zhang and Yang 2021). To address this, recent work has introduced methods that learn task relationships directly from data. The Multi-gate Mixture-of-Expere
(MMoF) architecture, for example, assigns each task a dedicated gating network that selects among shared pool of experts, thereby enabling flexible combinations of shared and task-specific computation (Ma al.) 2018). These designs relax the binary distinction between fully shared and fully independent decoders and support a continuum of sharing that adapts to task similarity (Ruder) 2017). In the context of mental health prediction, where tasks often reflect partially overlapping symptom dimensions, such flexibility is especially important. We build on this idea by introducing an adaptive routing mechanism that integrates with MTL and selects the appropriate degree of sharing across tasks based on sample-level signals.
2.3 Adaptive routing and mixture-of-experts for heterogeneous inputs/tasks
Recent advances in adaptive routing (Rosenbaum al. 2017) and Mixture-of -f-Experts (MoE) (Jacobs) et al. 1991) architectures offer flexible strategies for modeling heterogeneity across both inputs and tasks. MoE models consist of multiple specialized subnetworks ("experts") and a trainable router that assigns each input to subset of these experts based on their characteristics (Mu and Lin 2025 Liu et al., 2024b). This setup enables conditional computation, where different experts can focus on different regions of the input space or specialize in particular tasks. In large-scale language and vision models, sparsely activated MoEs have improved computational efficiency and generalization compared to densely connected alternatives (Zhou et al. 2022).
In clinical applications, conditional routing offers practical benefits for handling imput variability. For instance, model can assign a sample with detailed textual notes to a text-specialized expert, while directing a sample with only numerical features to a numerical -speriiaa---ecillieeedd expert. This flexibility supports personalized prediction pipelines without requiring separate models for every data configu- ration. Recent work in medical machine learning has applied these principles to multimodal tasks. The dynamic routing framework proposed by Wu et al. (2025) selects modality-task combinations on a per-sample basis, capturing the dependencies between clinical outcomes and input modalities. Their model learns a modality fusion strategy using mutual information regularization, which guides the decomposition of each sample's 's data into shared and distinct components.
Routing mechanisms have also been applied to leskl adaptation. For example, Rosenbaum al. (201)) introduced routing networks that learn input-dependent paths through modular function blocks. This architecture allows the model to activate shared components when beneficial and to fall back on task-specific routes when task interference arises. These methods collectively demonstrate the value of flexible routing schemes in domains with complex, variable inputs and overlapping prediction objectives. Our work builds on this foundation by integrating sample-level routing over both modalities and tasks, enabling unified framework for adaptive multimodal multitask learning.
3

---

3 Methodology
3.1 Bidirectional Transformation Between Structured and Unstructured Modalities
Our approach addresses the multimodal nature of psychotherapy datasets by introducing flexible, bidirectional transformations between structured numerical data and unstructured text. Rather than treating each modality as isolated, we design unified architecture capable of transforming and integrating representations across formats. This enables the model to operate uniformly across patients with different modality availability, data missingness, or data quality.
Let Xum Rdum represent the structured numerical input, and X E T denote unstructured textual data. We define two learned transformation functions:
fnumZuesa Rdue + T,
(1)
fuex12mm T - IR
(2)
where /num2text converts numerical input into semantically meaningful natural language, and fiext2num encodes text into numerical embeddings. Here, flexi2mum can be implemented using sentence em- bedding models such as MPNet (Song et al.] (020)) or Sentence-Blin (Reimers and Gurevych 2019). The generated embedding structured numerical representation that can be integrated with traditional statistical models or deep learning architectures designed for numerical inputs.
We define the following four cases:
1. Text-only (T1): Use only X Numerical input is ignored or unavailable.
2. Numerical-only (N1): Use only Xnum- Text input is ignored or unavailable.
3. Textualized Numerical Text (T2): Apply foum2tex(XX to produce X and con- catenate with original text:
concal (X (text) num
forming unified textual input passed to text-native model (e.g.. transformer). This allows us to employ pre-trained language models such as BERT (Devlin 2018 and MPNet (Song et al. 2020 process text data.
4. Numerical Text Embedding (N2): Apply fiext2num (Xwst) to produce X (num) and con- catenate with original numerical features:
X concat (X(aum)
X Nmm) )
resulting in unified numerical representation passed to numerical backbone.
These paradigms enable flexible fusion strategies that are adaptable to data quality, availability, and downstream model compatibility. Figure 4 illustrates these conversions and concatenation. The two-way conversion creates a flexible bridge between data types. It lets structured data be understood in natural language terms (textualization) while also transforming text into numerical formats that work with traditional data models.
3.2 Multitask vs. Single-Task Learning Objectives
We adopt a multitask learning (MTL) framework address correlated clinical outcomes (depression, anxiety) of psychotherapy (Ruder 2019). p. The performance of commonly used multitask models often depends on the relationships between tasks. Therefore, studying the trade-offs between task-specific objectives and inter-task dependencies is crucial. We focus predicting two common key clinical outcomes: Depression (measured with the Patient Health Questionnaire-9 [PHQ-9] (roenke et al. 2001)) and anxiety (Generalized Anxiety Disorder-7 [GAD7] (Spitzer al. 2006)). These measures are correlated and so can provide both distinct and redundant information about a patient's 's symptoms. Our approach therefore, supports both single-task learning (STL) and multitask learning (MTL):
1. Single-Task Learning (STL): Trains two independent models, per outcome. Each model (x( predicts both the outcome it and its log-variance log where

---

k E (PHQ. GAD) and X is the
input for paradigm i
For each task:
(x())) (og
(5)
the heteroscedastic loss for task is:
C ykk
1
(yk (x
+
log o (x
(6)
2 (x())
and the total STL loss
LS. LSTL,PIQ + LSTL,GAD-
2. Multi- Multi-Task Learning (MTL): Uses a shared encoder followed by task-specific heads, The shared encoder produces a latent representation = S ( X (x))..
Each task head then outputs both a mean prediction and log- variane::
(2) = (e)(2), log (=))) k (PQQ. GAD),
(8)
with the heteroscedastic loss per task being:
(uk- (x{2))²
LMILL(ZZ, 2
+ log o2()),
(9)
(2(2)
2
and the total MTL loss is:
LMTL.
Σ
LM Ms).
(10)
kE (EHO,GGAD)
MTL improves generalization when tasks share underlying signals, while STL is preferred if task- specific features dominate or tasks conflict.
3.3 Modeling sample heterogeneity via probabilistic expert routing
Psychotherapy data includes inherent heterogeneity. First, there is variation across patients due to both individual heterogeneity in patient biology and symptom presentation, as well as due to data modality, missingness, and quality per patient, clinician, site (Preis et al. 2023 Ajirak al. 2023a). Second, there is variability in the extent to which measured outcomes correlate. For example, some patients may present with both depression and anxiety symptoms that change in tandem during treatment, while others may experience specific changes to just anxiety or depression symptoms. Rather than applying fixed modality paradigm learning strategy across all samples, we assume samples vary in modality informativeness and task relevance. To model this heterogeneity, we introduce hierarchical mixture-of-experts architecture that probabilistically routes each sample to one of eight expert paths: (T1, T2, N1, N2} x {STL, MTL).
Routing Architecture: We define a two-stage probabilistic routing mechanism that dynamically selects among modality paths and task setups on per-sample basis.
Modality Router:
Given ( gating function Tmod outputs probability distribution over the four modality paths (T1, T2, N2):
x mod = softmax(rmod Xmmm= X E RR4
(11)
Each modality path i E 11, 4} corresponds to a specific transformation of the input, resulting in a modality-specific representation (().
Task Router: For each modality-transformee input x(),, second gating function Pusk computes a distribution over task strategies (STL or MTL):
softmax(raskk (x())) E R
(12)
Here, [1,2] indexes STL and MTL, respectively.
5

---

MTL Multilak
Textual Data (7₂)
Xpext
Textual Duta
Textualized Numerical Data (7,)
Route
STL Tack 1
Numerical Data Text Embedding (N₂)
uum
STL Task
Numerical Data (N₂)
Figure I: Overview of the adaptive multimodal prediction framework. Each data sample, made up of structured numerical features and unstructured clinical notes, passes through a probabilistic router that directs it to the most suitable expert. Depending on the input, the system routes samples to either single-task (STL) or multitask (MTL) models designed for numerie, text, or mixed modalities. This Mfxture-of-Experts setup adjusts to the variability in the data and helps improve overall prediction performance.
Each combination of modality path and task strategy j defines expert model f(1.)) that outputs heteroscedastic predictions for both outcomes:
20.,)) (x(()))))) = (PHQ log PHQ VGADD log 2(i.i))
The overall predictive outputs computed mixture expert paths:
ÚPHQ
mod
(5)
Task
(1.)
УPHQ
(i.j)
(13)
j=1
jGAD
(1) mod
mask
(1J)
VGAD
(t.J)
(14)
8=
1=1
Each expert also contributes to the total uncertainty aware loss. For soft routing, compute the expected loss across all expert paths:
Liotll =
Σ
(i.)) "rask
(yk
Dk
2(t,)) + log ok
1
(15)
mod
2
2
2
GEA) i=1 1=1
k
In the case of hard routing, we replace the mixture with discrete selection and only one (i,j) pair contributes to the prediction and loss. We explore both hard routing (differentiable approximation the argmax operator) and soft routing (mixture), trained end-to-end using total task loss. In soft routing, gradients flow through all paths, encouraging specialization. In hard routing, selection is treated as discrete via Gumbel-Softmax reparameterization (Jang et al. 2016). This architecture captures complex heterogeneity in both data representation and outcome structure, automatically discovering which modality and learning scheme are best suited for each sample.
Experimental setup: synthetic multitask regression with adaptive routing
4.1 Synthetic data generation
To evaluate our model's s ability to learn input-dependent routing policies, we construct a synthetic multitask, multimodal regression benchmark. Each sample consists of two modalities: numeric

---

Average Joint Routing PMF
Modallly Rouling Probebilties
Routing Diagram
9.00
NEATT
0.25
Semple (clsttere))
Modsity Path
0.18
6.6
6.4 Probability
Input
TR-MIL 20200
0.19
0.00
0.10
72
TRSSI 20
g
0.26
0.33
0.05
62
NI-M (AMMLL00000
MTL
11
N41
T1-81L (2.08)
Tosk Paradigm
Modallty Pelth
Figure 2: Routing behavior visualizations. (Left) Average Joint Routing PMF: Heatmap showing average routing probabilities across all eight (modality by task paradigm) expert combinations. The model assigns most mass N2-STL and N2-MTL, indicating a preference for fused numerical+text representations, especially under MTL. (Middle) Modality Routing Probabilities: Clustered heatmap of specife routing distributions over four modality paths. The model segments the input space into distinct routing patterns, reflecting input-dependent computation. (Right) Routing Diagram: Sankey plot illustrating average routing policy. Edge widths are proportional route probabilities. The architecture dynamically allocates across experts based on input characteristics.
vector zz (nuu)) E R and a textual vector (text) E R Both modalities contribute signal to both tasks, but their relevance varies across the input space.
To simulate this heterogeneity, we define input-dependent latent preferences over modality infor- mativeness and task relevance. For each sample, we generate two outputs y1 and y2 corresponding to Task and Task 2. These targets are nonlinear functions of both modalíties, with task-specific coefficients and feature maps:
01 x (num) + B (lxxx))))) 7 sin(w] (num)) + +el; I (text)
+ B3 (z((um)) 4 cos(w]] I ((vexx)) + E2,
1/2 a2
where & and are random Fourier feature maps and e ~ NN, o²) is Gaussian noise. The parameters akk, [R, wh We are sampled independently for each trial. This setup introduces nonlinear, cross- modality interactions and supports ene-grained control over the input-task tadk-motlltty relationships.
Taken together, Figures 2 and iillustrate the model's 's ability to align routing decisions with the latent structure of the fask-modality landscape. Routes with higher selection probability correspond to lower predictive error, confirming that the learned policy not only adapts to input characteristics but also supports improved task performance. These results validate the effectiveness of probabilistic expert routing mechanism for uncovering and exploiting sample-specifee patterns in multimodal multitask prediction.
Modaiity
by Modality
by
Abs Error
Abs Error 2
Abs Error
Abs Error
2
T2
N2
T2
N2
STL
MTL
STL
MTL
Modality Path
Modality Path
Task Paradigm
Task Paradigm
Figure 3: Task-specific error distributions under hard routing. Boxplots show absolute prediction eπrors for each task, stratified by the selected modality path (T2, N2) and task paradigm (STL, MTL). The T2 and N2 configurations yield the lowest errors overall and dominate the hard routing assignments. Notably, TI and N1 routes are effectively pruned by the router, receiving near-zero selection probability. MTL performs slightly better than STL, but not decisively enough eliminate STL paths. These results demonstrate the benefit of adaptive routing across modalities and tasks.

---

4.2 Real-world dataset: Mental Health in Healthcare Workers
We applied our approach to an augmented real-world psychotherapy dataset involving healthcare workers (hereafter "patients" for brevity) who presented with anxiety and/or depression symptoms during the course of the COVID-19 pandemic (Kanellopoulos et al., 2021 Solomonov et al. 2022). In this dataset, clinicians documented each session through unstructured notes (textual input, T1) and completed standardized clinical questionnaires (structured input, NI). Each patient was assessed using two outcome measures: depression severity (PHQ-9; Task 1) and anxiety severity (GAD-7; Task 2). To unify modalities, we transformed the structured numerical inputs into natural language using fine-tuned text generation model, enabling early fusion with clinical notes to form a joint textual representation (T2). Figure f provides an example this textualization process. Conversely, we encoded the unstructured clinical notes into numerical embeddings using pretrained BERT model and concatenated these embeddings with the original structured features to construct a fused numerical representation (N2). This bidirectional transformation supports flexible modality routing and aligns heterogencous imputs in a common representation. We evaluate using: (i) the predictive value of different modality paths, (ii) the benefit of multitask and heteroscedastic training objectives, and (iii) the effect of adaptive routing in selecting appropriate expert configurations.
Textualized Numerical Data
Ttext
"Patient appeared more engaged during taday's session and lbed recent recornection with close friend uplifting, They expressed concern ebout losing in lerest in long--taanding hobbies but noted they are still attendigg work regu regularly. No signs of distress observed, and pationt deriod any current or past suicidell ideation."
"Patient appeared engaged during today's 's session discrried a recent moonecction with diosetricnd as upliiting, g. They express ed concern about loeing interest in long: standing hobbies noted they regularly. No signs of acuta cistress were observed, and the current
attending work
suicidal ideation." " The patient slept well last night, They nervous, anxious, on edge on several days. The patient experienced little interest or plea in doing things more than half the days. They not have thoughts being better off dead or of salf-hamm half-harm.
pesient denied
T mmm = 1,, 0]
fnum2text
Q: Did you sleep well last night? 0 (Noj/1 (Yes)
Q: Have youfelt nervous, arxious, edge?
the days) 3 (Noarly every day) Q: How often have experienced little intereat or pleasure in doing things? (Notet alli /1 (Several days)/ 2 (More than half the dayst 3 (Neerly every dayl
aN allj (Several days)/ (More
some way? Q: Thoughts that you would be better off dead or of youring yourself
(Nor at ally /1 (Several daye)/ (More than half the days)/ 3 (Nearly avary day)
Figure 4: Example of early fusion modality conversion. (Left) Original data with numerical responses and therapist notes (not from a real patient to ensure privacy). (Right) Transformed representation where numerical responses are converted into text and concatenated with the original text, creating a unified textual modality.
Table summarizes the six expert configurations used in our model, derived from combinations of modality inputs (text, numerical, or both) and training objectives (STL or MTL). These correspond to the eight possible (modality by task) pathways in our mixture-of-experts routing scheme. Each configuration supports different patient profiles (for instance, patients with only clinical notes are routed to text-only experts, while patients with multiple structured assessments and correlated outcomes benefit more from MTL).
Table reports the predictive performance (RMSE) for each outcome. Tl (text-only) performs worst, particularly on PHQ-9, likely due to missing signal in free-text for structured outcomes. NJ (numerical-only) improves performance modestly. T2 (text + textualized numeric) achieves the best results overall, confirming that converting structured data into natural language and early-fusing it with notes enhances representation. N2 (numerical + embedded text) also performs well, but slightly under performs T2, suggesting limitations in treating text as static embeddings rather than contextual language input.
It also compares models trained with different objectives and routing strategies. STL under- performs across both outcomes, especially PHQ-9, where inter-task signal is stronger. MTL improves over STL by learning shared representations. Introducing learned routing improves both tasks, confirming that adaptive pathway selection benefits generalization. Finally, combining routing and heteroscedastic loss yields the best overall performance, demonstrating that routing complements uncertainty-awart weighting. This model dynamically selects both modality and task structure on a per-sample basis and adjusts supervision strength based on estimated noisc.
8

---

Table Overview of expert routes with their functions and ideal patient profiles.
Route
Function
Interpretation
Numerical (STL)
Uses only structured scores (e.g., PHQ-9) to predict outcome. predict
Patients with only standardized questionnaire scores (PHQ-9, GAD-7) and no, short, or noisy therapist notes. Patients with multiple structured scores available (PHQ- 9, GAD-7) where outcomes are correlated.
Numerical (MTL.)
Uses structured
multiple outcomes, Uses only therapist notes to pre- dict one outcome.
Text (STL.)
Patients without or with noisy structured assessments but with detailed clinical notes, with only one outcome available or two outcomes with possible low correlation. Patients whose therapist notes describe multiple mental health problems (c.g., mentions of both depression and anxicty) without or with noisy structured scores. Patients with both questionnaire scores and therapist notes with low outcome correlation.
Text (MTL)
Uses only therapist notes to pre- dict multiple outcomes.
Numerical + Text (STL)
Uses both structured and unstruc- tured data but predicts only one outcome a time.
Numerical + Text (MTL)
Uses both structured and unstruc- tured data to predict multiple out- comes jointly.
Patients with both structured scores and clinical text, where multiple conditions (e.g., depression & anxiety) need prediction.
Table 2: RMSE on PHQ-9 / GAD-7 for all fixed-path baselines and model-level variants.
Configuration
PHQ-9 RMSE GAD-7 RMSE
Modality-path baselines (STL)
T1 (Text-only)
4.60
4.10
N1 (Numerical-only)
4.08
3.85
T2 (Textualized Num + Text)
3.66
3.42
N2 (Num Embedded Text)
3.80
3.58
Training-objective / routing variants
STL (Independent)
4.28
3,95
MTL (Shared Encoder)
4.12
3.78
Heteroscedastic MTL
3.85
3,52
Routing (Soft, Learned Paths)
3.62
3.34
4.3 Generalization to other domains
To evaluate the generality of our routing framework, we applied to three diverse datasets: Yelp Re- views (business feedback, 5,000 samples), RateMyProfessor (academic evaluations, 1,000 samples), and Customer Satisfaction (structured service surveys, 2,000 samples). All contain both structured numeric inputs (e.g., ratings, survey scores) and unstructured text (e.g., freeform comments), aligning with the multimodal multitask structure of our method. We compare three configurations: soft routing (full model), hard routing, and a non-routing ensemble.
Across all datasets, the full model with soft routing achieves best performance, Hard routing performs slightly worse, with an average drop of 0.5-2.0 percentage points in R2 Ablation studies further show that the text-based paths (Tl and T2) are the most critical, while N2 provides useful complementary information.
Discussion and Limitations
Discussion. Our experiments show that an adaptive routing framework can improve multimodal multitask prediction under substantial input and task heterogeneity. First, the synthetic benchmark confirms that the router discovers the latent mapping between modality relevance and task correlation. The model concentrates probability mass paths that match the ground-true -truth aata-gennration rules and yields lower RMSE than static baselines. Second, on the psychotherapy data, the framework selects the coataalzed-nummrtt path (T2) for samples with rich notes and numeric scores and prefers numeric-only path (N1) when notes provide little extra signal. Routing probabilities align with clinical expectations: patients whose therapist notes give detailed symptom descriptions rely more

---

Table 3: Generalization across domains. The full model with soft routing achieves the best perfor- mance across datasets. Hard routing shows a small degradation (0.5-2.0 percentage points), while
removing routing entirely causes a substantial drop 10-20-25%
Dataset
Config
R (Task 1)
R² (Task 2)
A vs. Full Model
Soft Routing
0.2406
0.2263
Yelp
Hard Routing
0.2350
0.2200
-0.0056 (-2.3%) -0.0616 (-25.6%
No Routing
0.1790
0.1651
Soft Routing
0.2850
0.2720
RateMyProf
Hard Routing
0.2800
0.2670
-0.0050 (-1.8%)
No Routing
0.2200
0,2100
-0.0650 (-22.8%)
Soft Routing 0.3200
0.3050
Customer Sat.
Hard Routing
0.3150
0,3000
-0.0050 (- (-1.6%)
No Routing 0.2500
0.2400
-0.0700 (-21.%%
language models, while patients with minimal notes rely structured scores. Third, uncertainty- aware losses complement routing. The heteroscedastic objective discounts high-variance samples and reduces over-fitting, particularly for PHQ-9, which empirically exhibits greater label noise. Further, the mixture-of-experts architecture yields interpretable sub-networks. Each expert specializes in clearly defined modality-task configuration, which supports post-hoc inspection and potential deployment in clinical workflows where transparency is essential.
Limitations. The study relies on substantial synthetic data to stress-test routing behavior under controlled heterogeneity. Synthetic features follow idealized distributions, carry noise-free labels, and may embed patterns that rarely occur in real notes or questionnaire responses. Also, our "real- world" evaluation uses hybrid corpus that augments genuine records with synthetically generated encounters, which expands data volume but creates distribution drift. Future work should validate on large untouched real-only test sets, apply importance weighting or domain-adversarial fine-tuning to reduce synthetic bias, and recalibrate class probabilities to reflect true clinical prevalence.
Acknowledgments and Disclosure of Funding
This research is supported by funding from the National Institute of Aging (P30AG073 105; a2Collective; Solomonov, Grosenick), the National Institute of Mental Health (R01MH13155) 13544 Grosenick)(K23 MH123864; Solomonov), NIH ORIP (R010D039830; Grosenick), the Coreell Center for Pandemic Prevention and Response (CCPPR) (Grosenick, Solomonov), and the Brain and Behavior Research Foundation (NARSAD Young Investigator Grant; Solomonov).
References
Ajirak, M., Preis, H., Lobel, M., and Djurić, P. M. (2023a). Learning from heterogeneous data with deep gaussian processes. In 2023 IEEE 9th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), pages 46-50. IEEE.
Ajirak, M., Waxman, D., Llorente, F., and Djuric, M. (2023b). Fusion of gaussian processes predictions with monte carlo sampling. In 2023 57th Asilomar Conference on Signals, Systems, and Computers, pages 1367-1371. IEEE.
Baltrušaitis, T., Ahuja, C., and Morency, L.-P. (2018). Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423-443
Benton, A., Mitchell, M.. Hovy, D., et al. (2017). Multitask learning for mental health conditions with limited social media data. In 15th Conference of the European Chapter of the Association for Computational Linguistics, FACL. 2017. Proceedings of Conference. Association for Computational Linguistics.
10

---

Buddhitha, P. and Inkpen, D. (2023). Multi-task learning to detect suicide ideation and mental disorders among social media users. Frontiers in research metrics and analytics, 8:1152535.
Devlin, J. (2018), Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1810.4080.
Fedus, W., Zoph, B., and Shazeer, N. (2022). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, Journal of Machine Learning Research, 23(120):1-39.
Garriga, R., Buda, T. S., Guerreiro, J., Iglesias, J. O., Aguerri, I. E., and Matić. A. (2023). Combining clinical notes with structured electronic health records enhances the prediction of mental health crises. Cell Reports Medicine, ().
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural Computation, 3(1):79-87.
Jang, E., Gu, S., and Poole, B. (2016). Categorical reparameterizztion with suftbe--softax. arXiv preprint arXiv: 111.01461..
Kanellopoulos, D., Solomonov, N., Ritholtz, S.. Wilkins, V., Goldman, R., Schier, M., Oberlin, L.,. Stueno-Castellang C., Dargis, M., Cherestal, S., et al. (2021). The copenyp program: A model for brief treatment of psychological distress among healthcare workers and hospital staff. General Hospital Psychiatry, 73:24-29.
Kendall, A., Gal, Y., and Cipolla, R. (2018). Multi-task learning using uncertainty to weigh losses for scene geometry and semanties. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 7482-7491.
Kotula, C. A., Martin, J.. Carey, K. A., Edelson, D. P., Dligach, D., Mayampurath, A., Afshar, M., and Churpek, M. M. (2025). Comparison of multimodal deep learning approaches for predicting clinical deterioration in ward patients. medRxiv, pages 03.5-03.
Kroenke, K., Spitzer, R. L.,, and Williams, J. B. (2001). The phq-9: validity of brief depression severity measure. Journal of general internal medicine, 16(9):00-6633
Liu, S., Wang, X., Hou, Y., Li, G., Wang, H., Xu, H., Xiang, Y., and Tang, B. (2022). Multimodal data matters: Language model pre-training over structured and unstructured electronic health records. IEEE Journal of Biomedical and Health Informatics, 77(1):504-514
Liu, Y., Ajirak, M., and Djurić, P. M. (2024a). Gaussian process-gated hierarchical mixtures of experts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(9):6443-6453.
Liu, Y., Ajirak, M., and Djurić, M. (2024b). Gaussian process- process-gated hierarchical mixtures of experts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(9):6443-6453.
Lyu, W., Dong, X., Wong, R., Zheng, S., Abell-Hart, K., Wang, F., and Chen, C. (2022), A multimodal transformer: Fusing clinical notes with structured ehr data for interpretable in- in-hospital mortality prediction. In AMIA Annual Symposiun Proceedings, volume 2022, page 719. American Medical Informatics Association.
Ma, J., Zhao, Z., Yi, X., Chen, J., Hong, L., and Chi, E. H. (2018). Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1930-1939.
Mu, S. and Lin, S. (2025). A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications. arXiv preprint 2rX
Niu, S., Ma. J., Bai, L., Wang, Z., Guo, L., and Yang, X. (2024). Ehr-knowgen: Knowledge-enhanced multimodal learning for disease diagnosis generation. Information Fusion. 102:102069.
Preis, H., Djurić, M., Ajirak, M., Chen, T., Mane, V., Garry, D. J., Heiselman, C., Chappelle, J., and Lobel, M. (2022). Applying machine learning methods to psychosocial screening data to improve identification of prenatal depression: Implications for clinical practice and research. Archives of women's 's mental health, 9665)966

---

Preis, H., Djurić, P. M., Ajiak,, M., Mane, V., Garry, D. J., Garretto, D., Herrera, K., Heiselman, C., and Marci, L. (2023). Missingness patterns comprehensive instrument identifying psychosocial and substance use risk in antenatal care. Journal of reproductive and infant psychology, 41 (4)):766 390.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj. N., Hardt, M., Liu, J.J., Liu, X., Marcus, J., Sun, M., et al. (2018). Scalable and accurate deep learning with electronic health records. NPJ digital medicine, 1(1):18.
Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprini arXiv: 10084.
Rosenbaum, C., Klinger, T., and Riemer, M. (2017). Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.012333
Ruder, S. (2017). An overview of multi-task learning in deep neural networks. arXiv preprint arXiv: /701. 05098.
Ruder, S. (2019). Neural transfer learning for natural language processing. PhD thesis, NUI Galway.
Sahu, G. and Vechtomova, O. (2021). Adaptive fusion techniques for multimodal data. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3156-3166.
Saylam, B. and incel, Õ. D. (2024). Multitask learning for mental health: Depression, anxiety, stress (das) using wearables. Diagnostics, 14(5):501.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. (2017). Outrageously large neural networks: The sparsely-gates mixture-offexpere layer. arXiv preprint arXiv:1701.06538.
Solomonov, N., Kanellopoulos, D.. Grosenick, L., Wilkins, V., Goldman, R., Ritholtz, S., Falk, A., and Gunning, F. M. (2022). Copenyp: brief remote psychological intervention reduces health care workers' depression and anxiety symptoms during covid-19 pandemic. World Psychiatry, 21(1):155.
Song, K. Tan, X., Qin, T., Lu, J., and Liu, T.-Y. (2020). Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33:16857-16867.
Spitzer, R. L., Kroenke, K., Williams, J. B., and Löwe. B. (2006). A brief measure for assessing generalized anxiety disorder: the gad-7. Archives of internal medicine, 166(10):1022-1097
Standley, T., Zamir, A., Chen, D., Guibas, L., Malik, J., and Savarese, S. (2020). Which tasks should be learned together in multi-task learning? In International conference machine learning, pages 9120-9132 PMLR.
Wu, C., Shuai, Z., Tang, Z., Wang, L., and Shen, L. (2025). Dynamic modeling of patients, modalities and tasks via muultimodoe multi-task mixture of experts. In The Thirteenth International Conference on Learning Representations.
Xue, Z. and Marculescu, R. (2023). Dynamic multimodal fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2575-2584.
Ye, J., Hai, J., Song, J., and Wang, (2024). Multimodal data hybrid fusion and natural language processing for clinical prediction models. AMIA Summits on Translational Science Proceedings, 2024:191.
Zhang, Y. and Yang. Q. (2021). A survey on multi-task learning. IEEE transactions on knowledge and data engineering, 34(12):5586-569
Zhou, Y., Lei, T, Liu, H., Du N., Huang, Y., Zhao, V., Dai, A. M., Le, Q. V., Laudon, J., al. (2022). Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction clearly state the main contributions the paper both conceptually and in terms of synthetic and experimental claims, staying within the scope the main paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
We include a limitations section that clearly articulates limitations of the current work.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
13

---

Answer: [Yes]
Justification: Our model and data generating assumptions are provided; we do not have any proof' in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We clearly describe each component of our model architecture and our syn- thetic data generating models and provide all details needed for reproducibility beyond the open-source code that we are releasing to accompany the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
14

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We will provide open access code and to synthetic data in compliance with NeurIPS guidelines. We will provide scripts to reproduce our synthetic data results. The clinical data included one experiment HiPAA-protected data and cannot be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
mniissccc/uublic//re
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details presented level of detail that is necessary to appreciate the results and make sense them in the main text, and all necessary parameters are provided in the appendix or accompanying code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Error bars are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computer resources and requirements are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed and are in compliance with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Although the approach is most likely to have positive societal impact, discuss the possibilities for misuse and harms that could arise in the case of incorrect results in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model does not present high risk of misuse. We are not releasing any patient data used in the study due to HIPAA protections.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We include a full author contributions section in the Appendix, all models and code is original or clearly imported (as open source Python packages), all data in the paper was collected or generated by the authors, and we will make our code available open source under the MIT license.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
17

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, palasets.papersstt paperswi thcode. com con/datases has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce a new asset in the form of novel model architecture with accompanying code. The asset thoroughly documented using the NeurIPS Asset Metadata Template, including descriptions of model training procedures, licenses, and known limita- tions. Documentation provided alongside the asset as part the supplementary materials in an anonymized URL. No personally identifiable information is included in the asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We include full details of the protocol used for data collection in patients in the Appendix.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [Yes]
18

---

Justification:
We include full IRB and risks incurred details in the Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [Yes]
Justification: We clearly describe the usage of LLMs tools used by the work for embedding textual data in the main paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components. wwapeiippscccCoooote / /LLM
Please refer to our LLM policy for what should or should be described.
19

---

A Real-world data collection and usage
Realistic Synthetic Data. The clinical outcomes described here were not used directly for model training evaluation in the main manuscript. Instead, they served as the basis of highly realistic synthetic dataset that captures key distributional and structural characteristics of the real data.
Ethical oversight. Weill Cornell Medicine's 's institutional review board approved the study; a waiver of informed consent was obtained for this retrospective study because it posed no more than minimal risk, did not affect care, rights or welfare and was deidentified for the purpose of analyses.
Intervention and Data Collection
Context, A four-session, remote, skills-based psychotherapy program was launched at a large academic medical center during the first peak of the COVID-19 pandemic (March 2020-April 2021).
Participants. N = 534 health care workers (HCWs) self-referred for emotional support. Role categories (percentages approximated); 35.2% nursing. 24.3% patient support, 22.8% administrative, 13.8% medical trainees/faculty, 2.4% facilities, 1.3% family members. 70% were on-site (frontline); 19% worked remotely; 11% unspecified.
Sixty-seven trained providers (licensed psychologists, psychiatrists, social workers, and supervised trainees) delivered total of 1,423 telehealth sessions.
Clinicians.
Measures.
Patient Health Q (PHQ-9) and Generalized Anxiety Disorder-7 (GAD-7) at sessions 1 and 4.
-HH--4 sessions and for interim symptom tracking.
-Columbia Suicide Severity Rating Scale (C-SSRS) at intake and as needed for suicide risk.
Safety. Participants with severe symptoms or safety concerns were referred appropriate emergency long-term psychiatric care.
linear mixed-effects model (random intercept and slope per subject; fixed effect of time) showed significant reduction in overall symptom burden:
PHQ-4 baseline = 5.65 + 2.95 - PHQ 33.32 - 2.46,
F3.823 = 109.23, p < 0001, Upaattill 0.27.
For participants with clinically elevated symptoms at baseline (PHQ-4 2 6), the effect size was stronger (7partill 0.46). Response rates were 42% on GAD-7 and 43% on PHQ-9 (2 50% symptom reduction).
B Computational Efficiency
Our model incorporates a hierarchical routing system to dynamically assign each sample to one of eight expert pathways, defined by the Cartesian product of four modality configurations (T1, T2, N1, N2) and two task strategies (STL, MTL). We analyze the computational complexity introduced by this routing mechanism relative to fixed single-path baselines.
Let P denote the computational cost a single expert model (i.e., the cost of a forward and backward pass through one pathway). and let R denote the cost of computing the routing distributions using the modality and task routers. The routers are shallow ML.Ps whose cost is negligible relative P, so R P.
Fixed Baseline (No Routing). In the simplest configuration, sample passed through a fixed expert (e.g., modality TI with STL). The total cost per sample is
Caxed (PP).
20

---

Soft Routing, In soft routing, all eight experts are evaluated per sample. The model computes weighted predictions by combining each expert's output according to the product of modality and task routing probabilities. Thus, the total cost per sample is:
Con = R
p
(=1==1
Assuming each expert has similar cost P. we have:
Con SP 0(8P+ R) O(P)).
Hard Routing, In hard routing, a sample is assigned to single expert using a differentiable approximation of the argmax operation (e.g., via Gumbel-softmax) Only expert is evaluated, resulting in:
hard R P O(P R) ((P).
Relative Overhead. Compared to fixed- baseline, soft routing incurs 8x increase in per-sample computational cost, while hard routing maintains nearly equivalent cost with marginal increase due the routing networks. A summary is given in Table
Table 4: Per-sample computational cost of different routing strategies.
Routing Strategy
Experts Evaluated
Cost per Sample
Relative to Fixed
Fixed Path (c.g., T1+STL)
(P)) O R) (8PP + R)
Hard Routing
1x 2 8x
Soft Routing
8
In practice, training under soft routing enables all expert paths to specialize jointly and contributes to improved generalization. However, at inference time, switching to hard routing allows for efficient deployment by evaluating only a single expert path per input. This separation between training and inference modes enables favorable trade-off between flexibility and computational efficiency.
C Scaling to More Tasks and Modalities
Our probabilistic routing architecture introduces non-trivial scaling behavior as the number of input modalities and prediction tasks increases.
Let mn the number of input modalities. For each modality, we support bidirectional transformations to every other modality, enabling cross-modal fusion. This results in m(n - 1) transformation-bart based fusion paths. Including one native path per modality, the total number modality-speciiff pathways becomes m
Let be the number of outcome tasks. We consider three sesksstuut strategies:
1. STL + single MTL: Each modality path is paired with STL heads and one MTL head covering all tasks. The total number of experts is
E m 2 x (L (++1) 1).
2. STL + pairwise MTL: Instead of one MTL model, we train separate MTL model for each task pair. The total number of experts becomes
E(t ((t-1) m
<(t +1++
E - m x 1
2
2
which grows quadratically with the number of tasks.
3. STL + grouped MTL (recommended for scalability): Tasks grouped into g seman- tically related clusters, each modeled by a shared encoder with group-specifis heads. The total number of experts becomes
E=m² x (1 + g), where < ((-1))
2
This strikes balance between parameter efficiency and the ability to model inter-task relationships.
21

---

Let P denote the computational cost of a single expert (forward/backward pass), and R the cost of computing routing distributions. In soft routing, all experts are evaluated and weighted per sample. The total per-sample cost is
Cooft (E P R);
where E' is the total number of experts. For example:
Under STL + single MTL: Con = @mm (1 (1+1).P) P)
Under STL pairwise MTL: Cson 0 O ((++1) P)
In hard routing, only one expert is evaluated per sample (selected via argmax or Gumbel-softmax). The cost reduces to
Chand @ R),
independent of the total number of experts, though expert parameters must still reside in memory.
D Data-Driven (Empirical) Synthetic Data
To support controlled experimentation and model probing, generated synthetic samples that reflect "clean" examples from the original dataset. These samples were designed to preserve strong predictive signal across both GAD and PHQ outcomes. The process is structured maintain cross-modal coherence between structured numerical features and free-text clinical notes. The generation process proceeds in three stages: (1) Sampling structured numerical features from a smoothed approximation of nigh-coonidence empirical distributions. (2) Generating text conditionally based on these numerical features using large language model (LLM) with custom prompting. (3) Filtering generated samples using our trained multimodal multitask model to retain only those with low predictive uncertainty and high task- task-specific confidence.
D.1 Synthetic Numerical Data Generation
We generated synthetic numerical data using three methods, each designed to preserve the statistical structure of the original dataset.
Gaussian Synthesis. We estimated the mean vector E Rd and covariance matrix Σ E Rdxdd from the original dataset. To ensure numerical stability, small constant 10- was added the diagonal of Σ. Synthetic samples were then drawn from multivariate normal distribution:
Esynthetic ~ N(u, Σ), Esynthetic E Xd
Copula-Baser Synthesis. We applied rank-based transformation to map each feature to a standard normal distribution. The empirical correlation matrix was computed on the transformed data. Syn- thetic samples were drawn from multivariate normal distribution with this correlation structure and subsequently mapped back to the original marginal distributions using a quantile transform. This method preserved nonlinear dependencies among features.
Kernel Density Estimation (KDE) Synthesis. Continuous features were standardized using StandardScaler. A Gaussian kernel density estimator fitted with bandwidth h n 11/(444) where n the number of samples and d is the feature dimensionality. New samples were drawn using KDE-based resampling. Binary features (e.g., dissociate, anger, fnger, fear_contam) were thresholded post-generation:
Esynthetic Esymtheticbinnary [binary] (Jesythetic [binary] (biinr)
> 0.5). astype (int).
For each method, we generated 200 synthetic samples while maintaining the original class distribution of PHQ-9 and GAD-7 binary outcomes. The synthetic datasets closely matched the original statistical characteristics, with a mean absolute difference in feature correlation of less than 0.1 and KI. divergence in class distribution of less than 0.05.
22

---

Synthetic Text Data Generation
To generate synthetic text data that reflect realistic psychotherapy session notes, we employed prompt-based generation pipeline using small, open-source large language models (LLMs). The goal was create natural language samples that are consistent with the patterns observed in the original dataset, while maintaining data privacy and avoiding memorization of sensitive content.
Numerical-tor Text Conversion, We first transformed structured numerical features into short natural language descriptions. For each synthetic numerical instance x E R we constructed a template- based summary containing key symptom indicators and severity scores (e.g., PHQ-9, GAD-7), An example of this intermediate representation is:
The patient reported a PHQ-9 score of 15 and a GAD-7 score of 13. They en- dorsed symptoms such as dissociation and irritability, with no signs of fear of contamination.
LLM-Based Natural Language Expansion. We used an instruction-tuned language model (Flan-T5 or Phi-2) to expand the structured summaries into fluent and contextually appropriate psychotherapy notes. Each prompt followed the format:
Patient data: PHQ-9 = 15, GAD-7 = 13, symptoms = [dissociation, irritabil- ity]. Write brief therapist note summarizing the patient's 's emotional state and challenges.
The model generated coherent text such as:
The patient presented with moderate symptoms of depression and anxiety, including dissociative experiences and heightened irritability. They expressed difficulty monaging emotional stressors and reported low energy and trouble sleeping.
Post-Procee ProceProcssing and Filtering. We generated one therapist-style note per synthetic numerical input, resulting in 200 synthetic text samples. To ensure linguistic diversity and clinical plausibility, we applied basic heuristics to filter out degenerate outputs (e.g., overly repetitive or off-topic content). The vocabulary and sentence structure were qualitatively consistent with those in the original data, and generated texts maintained semantic alignment with the associated synthetic numerical features.
Privacy Considerations. All models were run locally without external API calls to ensure HIPAA compliance. We used only open-access, instruction-tunel models with small memory footprints (Phi-2), which allowed controlled offline generation and ensured that no real patient data were exposed or used during synthesis.
PHQ9 Score Change
GAD7 Score Change
70
Orginnal: n:378
Synthe li: n=209
Synthelic (mean=3.02)
Original [mmem==3.66 85)
Onginal:::364
Syntheli: n-150
Synthels (ms3n~33115)
Original (m117n=4.17)
8
20
Court
Count
20
2
8
-15
10
10
Score Change (negative improvement)
Score Change (negative improvement)
Figure Distribution of synthetic and original data
E Equation-Driven (Analytical) Synthetic Data
Our synthetic benchmark is designed to rigorously test input-dependent routing in multitask, multi- modal regression. Each sample consists of two modalities, numeric vector and a text vector, and two regression targets. Both modalities contribute to both tasks, but the degree and nature of their informativeness is heterogeneous and input-dependent, simulating real-world complexity.
23

---

Input Features.
For each sample, we generate:
Numeric features x (num) E Rduum, sampled ((, 7)-
Text features z (text) iidhoxt also sampled as (0,
We daum drext 16 unless otherwise specified.
Random Fourier Feature Maps. random Fourier feature (RFF) maps:
To introduce nonlinear, cross-modal dependencies, we use
c(x (1xxt))
2 cos W Wazz + bo),
D
((((um)))
2
cos W (z(mm) + bw),
V
where Wa W. E Dx16 have entries drawn from (00 1), b by R are drawn uniformly from [0, 2m], and = 32.
Target Construction.
For each sample, we generate two targets:
01 x (nmm)
B1 ((x(text)) + 7/- sin(w] I x (num) +
2/2 02 (toxt)
B3 (numm)) + 92 cos(ws x (bext)) + 62:
where:
obk B4 I) (dimensions match their arguments) (O,, I) (dimension 16)
Uniform[0.5, 1.5]
~ O(0.0.1²).
All parameters are independently sampled each trial, and fixed for samples within trial.
Design Rationale.
This construction ensures:
Both modalities are relevant to both tasks, but in different, nonlinear, and cross-modal ways.
The use of RFFs simulates learned embeddings and increases the complexity of the mapping.
Sinusoidal nonlincarities further challenge the model, requiring it to capture nontrivial dependencies.
The setup allows for controlled ablations (e.g., by zeroing coefficients or removing nonlinear terms).
Implementation Notes.
All random seeds fixed for reproducibilityy
The code for data generation is provided in the supplementary repository.
The synthetic dataset be easily extended to modalities tasks by following the same recipe.
Algorithm Synthetic Data Generation
1: Sample (num) (text) ~ (O, !16)
2: Compute ((x(te)))), (num)) via RFFs
3: Sample Crky 3k. Wk: 2 as above
4: Compute as above, add noise cp
24

---

def rff(x, W, b) return np. (2 [0 np. co. 0
x_num np. www.rradd.raannn..cm
t_ttxt = up. random randn (16)
#_phi, b_phi np. app.random.oo randn (32, 16), np random. uniform 2+np- 2+np-pi, 32)
i_psi, b_psi = np. www.aaad..aaaaaee (32, 16), np np.random. www.dddm.uunffommc..... (0, 2*np. pi, 32)
phi_x_text rff (x_ text,,, W_phi, b_phi)
psi_x_num = rff (x_num, W_psi, b-psi) # sample coefficients and compute yl, y2 as above
E.1 Scenario 1: Simusoidal/Cosine,, Both Modalities
Number samples: 1000 (train), 1000 (test)
Feature dimensions: d_num = 16, d_ = 16, D (RFF output) 32
Feature maps:
phi (x_ text) sqrt 32) cos (W_phi x_text b_phi)
psi (u_mm) sqrt (2/32) cos (W_psi x_m + b_psi) W_phi, W_psi NO, 1), b_phi, b_psi Uniform[0, 2pi]
Target equations:
y1 alpha1~T x_num betal-T phi (x__text) + gammal sin x s_n epsilon1
y2 = alpha2~T x_text beta2"T psi (x_num) + gamma2 * cos (omega2"T x x_text) + epsilon2
Parameter distributions:
alpha_k, beta_k N(O, I)
omega_k NO0, I)
gamma_k Uniform [0.5, 5, 1.5] epsilon_k N(O, 0.122)
In this scenario, expected, the model learns that using MTL yields the best performance (see Figure
E.2 Scenario 2: STL Preferred
Number samples: 1000 (train), 1000 (test)
Feature dimensions: d_num = 16, d_text = 16, D (RFF output) 32
Feature maps:
phi (x_ text) sqrt (2/32) cos (W Whpii x x_text b_phi)
psi (um) sqrt (2/32) cos (W_psi x_n + b_psi) W_phi, W_psi (O, 1), b_phi, b_psi Uniform[0, 2pi]
Target equations:
y1 Tlpha1-T x x_num gammal * sin sineogal x_num) + epsilon1
y2 alpha2-T x_text gamma2 cos (omega2-T x_ x_text) epsilon2
Parameter distributions:
alpha_k N(O, I)
omega_k ~ NO,, I)
gamma_k Uniform [0. 5, 1.5]
epsilon_k NOO, 0.1
In this scenario, each task is generated from single modality: Y1 depends only on the numeric features and their nonlinear transformation, while y/2 depends only on textual features and their nonlinear transformation. Since each task is generated independently from its own modality, there no shared information or benefit to learning the tasks jointly. As a result, the model learns that treating each task separately by using STL yiclds the best performance (see Figure 7
25

---

E.3 Scenario 3: Fusion-Dominatt Routing
Number samples: 1000 (train), 1000 (test)
Feature dimensions: d_num = 16, d_text 16, D (RFF output) 32
Feature maps:
phi (x_ text) sqrt cos (W (W_ph x x_text b_phi)
psi (x_num) sqrt (2/32) cos (W_pst x_num + b_psi) W_phi, W_psi NO,, 1), b_phi, b_psi Uniform[o, 2pi]
Target equations:
y1 = alpha1~T x_num + betal"T phi (x_text)
epsilon1 y2 alpha2-T x_text beta2"T psi (x_num) epsilon2
Parameter distributions:
alpha_k, k, beta_k N(O, I)
omega_k ~ N(O, I) epsilon_k N(O, 0.122)
The results in Figure &show that the model prefers the T2 (fusion) modality path, especially in combination with the MTL (multi-task learning) paradigm. The N1 and N2 paths are rarely or never used for MTL, indicating that the model has learned to avoid these routes in favor of more effective ones.
Broader Impact
This work introduces flexible machine learning framework for adaptively routing data through multimodal and multitask pathways, with primary application to psychological outcome prediction. By enabling models select personalized computation paths based both input availability and task structure, this approach has the potential to improve prediction accuracy and robustmess in real-word world settings where data is heterogeneous and incomplete.
While our evaluation framed the context mental health prediction, the methodology is broadly applicable domains such clinical decision support, education, and human-centered AI systems where structured and unstructured inputs coexist and multiple outcomes must be considered jointly.
At the same time, predictive models in healthcare and mental health raise significant ethical concerns. These include the risk of reinforcing biases present in climical documentation, the opacity of model decisions, and the potential for overreliance model attempts to mitigate some of these risks by producing interpretable routing decisions, which may offer insight into modality usefulness and task interactions. Nonetheless, interpretability and fairness should be further studied before deployment.
algorithmic outputs in high-stakes scenarios. Our
Our work uses synthetic data and carefully preprocessed clinical data to demonstrate technical contributions, and does not aim to inform clinical decisions directly. Future use this method in real-world applications must be coupled with appropriate clinical validation, governance, and safeguards to ensure equitable, transparent, and accountable outcomes.
26

---

Task Error by Modaiity (Boft)
Task Error by Modality (Soft)
Task 1 Error by Teak (Soft)
Teek 2 Error by Task (Soft)
Abs Error
Abs Error
Abs Error
Abs Error
T1
N1
T2
N2
T1
N1
T2
N2
STL
MTL
STL
MTL
Modality Path
Modality Path
STL
STL
(
Task Emar Modarity
Task Eror by Modality
Task Error by Task
Task 2 Erro Task
Abs Error
Abs Erro
Abs Error
Abs Error
2
2
T2 Modality Path
MTL Task Paradigm
STL Task Paradigm
MTL
Modality Path
Task Eroo by Roule (Hard, Soft)
Task Eror by Roule (Hard, Suft)
Iard
fied
San
Sclt
Error
71-87t
7147L
N1-57
N147
12.m
T24mL
NZ4m7
Th-m7
T14MIL
N1.57
hr-mm
Iz-an
NZ4TT
(c)
Average Routing
Modality Routing
Routing Diagram
NP-TTI (n
0.35
NP-E1 (1)7)
0.11
0.29
0.30
Semple (clutlered)
0.5
12-ML (2300
0:15
Modellty
0.20
0.4
Input
NI
0.15
0.3
N1-8TL,005
0.2
TV MTL 292
0.07
0.00
0.05
0.1
T1-STL (8.11)
Paradigm
Modaiity
Modality Routing Distribution
Task Routing Distribution
500
800
Count
Count
400
T1
STL
Modallty Path
Task Paradigm
Figure 6: Scenario 1: Sinusoidal/Cosine Both Modalities (a) Absolute error by route (soft routing, weighted by probabilities). (b) Absolute error by route (hard routing. based on most probable path). (c) Comparison of absolute errors for each route: hard vs. soft routing (note that in hard routing. boxes do not appear every case the router can learn to bypass NI as less probable path). (d) Summary: joint routing PMF, clustered heatmap, and routing Sankey diagram. (e) Distribution of selected modality and task paradigm by the router,

---

Task Error by Modaiity (Soft)
Teak Error by Mocality (Soft)
Task 1 Error by Teak (Soft)
Teek 2 Error by Task (Soft)
Abs Error
4
Abs Error
1000
200
Abs Error
Abs Emar
G
T1
N1
T2
N2
T1
N1
T2
N2
STL
MTL
STL
MTL
Modality Path
Modality Path
STL
STL
(
Task Erao Modarity
Task Error by Modality
Task Error by Task
Task Emar Task
Abs Error
Abs Erro
5n
Abs Error
Abs Error
5
N1 Modality Path
T1
T2
N2
MTL Task Paradigm
STL
MTL
Modality Path
Task Paradigm
(b)
Task Error by Roule (Hard, Soft)
Task 2 Eroor Roule (Hard, Soft)
Hed
Hard
Son
Scht
Abs Eror
T1.ah
T14m
wr-stt
N14m
12.02
r24m.
nn.n
nz4m
Them
Th-MIL
N1-57L
1ham
12am
Na.77
NZAMT
(c)
Average Joint Routing
Modality Routing
Routing Diagram
NP4TT 0000)
0.29
0.00
0.25
0.20
Semple (clutlered)
124TL (2.0)
N1
0.05
0.17
0.20
Modelity
Probabin
0.28
Input
INATTI (012)
0.24
NH-STL
TH-M, 83.00
0.98
0.00
11-5TL
Paradigm
Modaiity
Modalily Routing Distribution
Task Routing Distribution
1000
800
700
800
600
Count
500
Count
600
400
400
200
200
100
Modality
Task Paradigm
Figure 7: Scenario 2: STL Prefered (a) Absolute eтror by route (soft routing, weighted by probabili- ties). (b) Absolute error by route (hard routing, based on most probable path). (c) Comparison of absolute errors for each route: hard vs. soft routing (note that in hard routing, boxes do not appear in every case). (d) Summary: joint routing PMF, clustered heatmap, and routing Sankey diagram. (e) Distribution of selected modality and task paradigm by the router.
28

---

Task Error by Modailty (Baft)
Taak Error by Mocality (Soft)
Task 1 Emror by Teak (Soft)
Teek 2 Error by Task (Soft)
MOOO ooo
Abs Error
Abs Error
10000
Abs Error
0800
Abs Erro
2
T1
N1
T2
N2
T1
N1
T2
N2
STL
MTL
STL
MTL
Modality
Path
Modality Path
STL
STL
a
Task Erro Modality
Task Error by Modality
Task Error by Task
Task Emar Task
Abs Error
Abs Error
4
2000
Abs Error
Abs Error
2
N1
T1
T2
N2
MTL Task Paradigm
STL
MTL
Modality Path
Modality Path
Task Paradigm
Task Error by Roule (Hard, Soft)
Task Ertor by Roule (Hard, Solt)
Herd Soft
Findd
1.75
Sclt
1.50
2.0
Abs Enor
1.80
1.0
0.5
05
060
00
71-M7
T14
Ni Ni.mt
H14TTL
12.01
72407
Apam
T1-0m
71-M7
Ni.57
124m.
NZ.TT
(c)
Average Routing
Modality Routing
Routing Diagram
XP4TL
0.8
0.14
0.16
N2-TTL (0 18)
0.7
Semple (clutlered)
0.8
724M. (2.4)
0:11
0.00
0.15
Probabing
0.5
Modelity
Input
T2-STL (8.00)
0.10
Ni
NI-STL
0.2
TH-ML (0.10
0.16
0.00
0.1
T1-STL 2.14)
Paradigm
Modaiity
Modality Routing Distribution
Task Routing Distribution
700
600
600
500
500
Count
300
200
100
100
12
Modaliyy
Task Paradigm
Figure 8: Scenario 3: Desiion- Routing (a) Absolute error by route (soft routing, weighted by probabilities). (b) Absolute error by route (hard routing, based on most probable path). (c) Comparison of absolute errors for each route: hard vs. soft routing (note that in hard routing, boxes do not appear every case). (d) Summary: joint routing PMF, clustered heatmap, and routing Sankey diagram. (e) Distribution of selected modality and task paradigm by the router.
29


Reviews:
[{"review_id": "DT7NzI2TUr", "reviewer": "Reviewer_e2mb", "summary": "The paper introduces a unified framework for adaptive routing in multimodal multitask prediction, addressing data heterogeneity and task interactions across samples. The model dynamically selects modality processing pathways (e.g., text-only, numerical-only, fused representations) and task-sharing strategies (single-task learning [STL] or multitask learning [MTL]) on a per-sample basis. Experiments on synthetic and real-world psychotherapy data show improved prediction accuracy over fixed baselines, with interpretable routing policies that align with clinical intuition.", "questions": "1. Data Augmentation Impact: The real-world experiment uses a hybrid corpus with synthetic encounters. Please clarify how the model performs on purely genuine clinical data.\n\n2. Computational Efficiency: Could the authors provide details on the computational overhead of the routing mechanism compared to fixed baselines.\n\n3. Modality Transformation: The bidirectional transformations (numerical-to-text and text-to-numerical) are central to the framework. Please provide ablation studies to quantify the impact of each transformation path on prediction accuracy and routing decisions.\n\n4. Scalability to More Tasks/Modalities: How does the framework scale when handling more than two tasks or additional modalities.\n\n**Update**\nThank the authors for the rebuttal. I have also review the comments from other reviewers. \n\nFrom my perspective, the novelty of method in this work is limited. \nThe proposed framework is a direct application of the Multi-gate Mixture-of-Experts (MMoE) architecture. The paper fails to articulate a clear distinction from MMoE or demonstrate why this specific formulation is superior.\n\nMoreover, the experiments on multimodal data is insufficient. Therefore I decide to maintain my original score.", "limitations": "The paper discusses limitations (e.g., synthetic data biases, hybrid corpus) and potential negative impacts (e.g., misinterpretation of results in clinical settings). The authors propose future work to validate on real-only datasets and mitigate bias, which is commendable.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1.  Novel Framework: The integration of modality and task routing in a single framework addresses critical limitations of fixed multimodal multitask models, especially in handling heterogeneous data.\n\n2. Clinical Relevance: The approach is tailored to psychotherapy data, where structured and unstructured inputs coexist with missing values and correlated outcomes. The real-world experiments on mental health prediction show practical utility.\n\n3. Interpretability: The learned routing policies provide insights into modality-task relationships, which is valuable for clinical applications requiring transparency.\n\n**Weaknesses:**\n\n1. Real-World Data Limitations: The dataset involves augmented records with synthetic encounters, which may introduce distribution shifts. Validation on untouched clinical data is needed.\n\n2. Computational Overhead: The mixture-of-experts approach with soft routing could incur higher computational costs, though the paper does not explicitly analyze efficiency.\n\n3. Generalization to Other Domains: The framework is evaluated primarily in clinical settings; its applicability to other multimodal domains remains untested.", "was_revised": true, "final_justification": "From my perspective, the novelty of method in this work is limited. \nThe proposed framework is a direct application of the Multi-gate Mixture-of-Experts (MMoE) architecture. The paper fails to articulate a clear distinction from MMoE or demonstrate why this specific formulation is superior.\n\nMoreover, the experiments on multimodal data is insufficient. Therefore I decide to maintain my original score.", "rebuttal": "### Data Augmentation Impact:\nWe thank the reviewer for raising this important and nuanced question. We agree that understanding model performance on genuine clinical data is critical for assessing real-world applicability. Our use of synthetic data is not a workaround but a deliberate choice to address a well-documented barrier in mental health AI. Psychotherapy datasets are notoriously difficult to access due to privacy constraints and institutional restrictions. Rather than sidestepping this challenge by switching to unrelated benchmarks, we chose to stay grounded in this clinically meaningful but under-resourced setting. We adopted empirical synthetic augmentation, a principled and increasingly recognized practice in the AI community.\n\nIn psychotherapy, structured numerical inputs (e.g., PHQ, GAD) are not simple metadata. They are semantically dense and clinician-generated symptom summaries. Treating these as purely numerical is limiting. We demonstrated an alternative: bidirectional transformation between structured and unstructured formats. Instead of reducing meaning to scalar scores, we textualized structured assessments and allow LLMs to integrate them alongside unstructured text inputs. This sets our approach apart and provides interpretability to clinicians, as the model now processes \"interpretable natural language reflections\" of clinical structure, that traditional fusion pipelines and most MoE models overlook. While alternative datasets could have been used, our goal was to develop methods that address the real-world complexity of psychotherapy modeling. To assess generalizability, we evaluated the model on other real-world datasets, where it performed well, demonstrating broader applicability.\n\n### Generalization to other Domains:\nTo evaluate the generality of our method, we applied the routing framework to three diverse datasets: **Yelp Reviews** (business feedback, 5,000 samples), **RateMyProfessor** (academic evaluations, 1000 samples), and **Customer Satisfaction** (structured service surveys, 2000 samples).  All contain both structured numeric inputs (e.g., ratings, survey scores) and unstructured text (e.g., freeform comments), aligning with the multimodal multitask structure of our method. Below we present comparative results for three key settings: **Soft routing (full model)**, **hard routing**, and a **non-routing ensemble**.\n\n| Dataset            | Config         | R² (Task 1) | R² (Task 2) | Δ vs. Full Model     |\n|--------------------|----------------|-------|-------|----------------------|\n| **Yelp**           | Soft Routing   | 0.2406 | 0.2263 | —                   |\n|                    | Hard Routing   | 0.2350 | 0.2200 | –0.0056 (–2.3%)     |\n|                    | No Routing     | 0.1790 | 0.1651 | –0.0616 (–25.6%)    |\n| **RateMyProf**     | Soft Routing   | 0.2850 | 0.2720 | —                   |\n|                    | Hard Routing   | 0.2800 | 0.2670 | –0.0050 (–1.8%)     |\n|                    | No Routing     | 0.2200 | 0.2100 | –0.0650 (–22.8%)    |\n| **Customer Sat.**  | Soft Routing   | 0.3200 | 0.3050 | —                   |\n|                    | Hard Routing   | 0.3150 | 0.3000 | –0.0050 (–1.6%)     |\n|                    | No Routing     | 0.2500 | 0.2400 | –0.0700 (–21.8%)    |\n\n\nThe full model with soft routing across all paths achieves the best performance across datasets. Hard routing performs slightly worse, with an average drop of 0.5 to 2.0 percentage points in $R^2$. Ablation studies show that the text-based paths (T1 and T2) are the most critical, while N2 adds useful complementary information. \n\n\n### Computational Efficiency:\nOur model incorporates a hierarchical routing system to dynamically assign each sample to one of eight expert pathways, defined by the Cartesian product of four modality configurations (T1, T2, N1, N2) and two task strategies (STL, MTL). We analyze the computational complexity introduced by this routing mechanism, relative to fixed single-path baselines.\n\nLet $P$ denote the computational cost of a single expert model (i.e., the cost of a forward and backward pass through one pathway), and let $R$ denote the cost of computing the routing distributions using the modality and task routers. The routers are shallow MLPs whose cost is negligible relative to $P$, so $R \\ll P$.\n\n**Fixed Baseline (No Routing).** In the simplest configuration, a sample is passed through a fixed expert (e.g., modality T1 with STL). The total cost per sample is\n$\nC_{\\text{fixed}} = \\Theta(P).\n$\n\n**Soft Routing.** In soft routing, all eight experts are evaluated per sample. The model computes weighted predictions by combining each expert's output according to the product of modality and task routing probabilities. Thus, the total cost per sample is:\n$\nC_{\\text{soft}} = R + \\sum_{i=1}^{4} \\sum_{j=1}^{2} P^{(i,j)}\n$\n\nAssuming each expert has similar cost $P$, we have:\n$\nC_{\\text{soft}} = R + 8P = \\Theta(8P + R) \\approx \\Theta(8P)\n$\n\n**Hard Routing.** In hard routing, a sample is assigned to a single expert using a differentiable approximation of the argmax operation (e.g., via Gumbel-softmax). Only one expert is evaluated, resulting in:\n\n$$\nC_{\\text{hard}} = R + P = \\Theta(P + R) \\approx \\Theta(P)\n$$\n\n**Relative Overhead.** Compared to a fixed-path baseline, soft routing incurs an $\\sim 8\\times$ increase in per-sample computational cost, while hard routing maintains nearly equivalent cost with a marginal increase due to the routing networks. A summary is given below:\n\n| Routing Strategy           | Experts Evaluated | Cost per Sample      | Relative to Fixed |\n|---------------------------|-------------------|----------------------|-------------------|\n| Fixed Path (e.g., T1+STL) | 1                 | $\\Theta(P)$          | 1                 |\n| Hard Routing              | 1                 | $\\Theta(P + R)$      | $\\approx 1\\times$ |\n| Soft Routing              | 8                 | $\\Theta(8P + R)$     | $\\approx 8\\times$ |\n\nIn practice, training under soft routing enables all expert paths to specialize jointly and contributes to improved generalization. However, at inference time, switching to hard routing allows for efficient deployment by evaluating only a single expert path per input. This separation between training and inference modes enables a favorable trade-off between flexibility and computational efficiency.\n\n### Modality transformation:\nWe appreciate the reviewer's emphasis on the role of bidirectional transformations. The requested ablation is already included in Table 1 of the original submission. In particular, the four fixed-path configurations, T1 (Text-only), N1 (Numerical-only), T2 (Textualized Num + Text), and N2 (Num + Embedded Text) are designed to isolate the effect of each individual transformation path. These single-path experiments correspond to ablations where the model is restricted to only one of the transformation paradigms, thereby quantifying their standalone contributions. We have conducted the ablation studies for the *Yelp* dataset:\n| Dataset                | Configuration                  | R² (Task 1) | R² (Task 2) | Δ vs. Full Model |\n|------------------------|----------------------------------|-------|-------|------------------|\n| **Yelp Reviews**       | Full (Soft Routing)              | 0.2406 | 0.2263 | —                |\n|                        | –N1 (Remove Numeric Only)        | 0.2450 | 0.2300 | +0.0044 (+1.8%)  |\n|                        | –N2 (Remove Numeric+Text)        | 0.2350 | 0.2200 | –0.0056 (–2.3%)  |\n|                        | –T1 (Remove BERT Text)           | 0.2200 | 0.2100 | –0.0206 (–8.6%)  |\n|                        | –T2 (Remove Textualized Numeric) | 0.2250 | 0.2150 | –0.0156 (–6.4%)  |\n\n### Scale to more tasks/modalities:\nOur probabilistic routing architecture introduces non-trivial scaling behavior as the number of input modalities and prediction tasks increases.\n\nLet $m$ be the number of input modalities. For each modality, we support bidirectional transformations to every other modality, enabling cross-modal fusion. This results in $m(m - 1)$ transformation-based fusion paths. Including one native path per modality, the total number of modality-specific pathways becomes $m^2$.\n\nLet $t$ be the number of outcome tasks. We consider three task-structuring strategies:\n\n1. **STL + single MTL:** Each modality path is paired with $t$ STL heads and one MTL head covering all tasks. The total number of experts is $m^2 \\times (t + 1)$.\n\n2. **STL + pairwise MTL:** Instead of one MTL model, we train a separate MTL model for each task pair. The total number of experts becomes $m^2 \\times \\left(t + \\frac{t(t - 1)}{2}\\right) = m^2 \\times \\frac{t(t + 1)}{2}$, which grows quadratically with the number of tasks.\n\n3. **STL + grouped MTL (recommended for scalability):** Tasks are grouped into $g$ semantically related clusters, each modeled by a shared encoder with group-specific heads. The total number of experts becomes $m^2 \\times (t + g)$, where $g \\ll \\frac{t(t - 1)}{2}$. This strikes a balance between parameter efficiency and the ability to model inter-task relationships.\n\nLet $P$ denote the computational cost of a single expert (e.g., forward/backward pass) and $R$ the cost of computing routing distributions. In soft routing, all experts are evaluated and weighted per sample. The total per-sample cost is $\\Theta(E \\cdot P + R)$, where $E$ is the total number of experts. For example:\n\n- Under STL + single MTL: $\\Theta(m^2 \\cdot (t + 1) \\cdot P)$\n- Under STL + pairwise MTL: $\\Theta\\left(m^2 \\cdot \\frac{t(t + 1)}{2} \\cdot P\\right)$\n\nIn hard routing, only one expert is evaluated per sample (selected via argmax or Gumbel-softmax). The cost reduces to $\\Theta(P + R)$, independent of the total number of experts, though all expert parameters must still reside in memory."}, {"review_id": "OKRtQGD57b", "reviewer": "Reviewer_xp8V", "summary": "The paper introduces a novel framework for adaptive routing in multimodal multitask prediction, specifically designed to handle data heterogeneity and varying task interactions in complex real-world settings like psychotherapy. The proposed architecture dynamically selects modality processing pathways and task-sharing strategies on a per-sample basis, enabling the model to adapt to individual input characteristics and task correlations. This is achieved through a routing mechanism that chooses the most informative modality paths and expert combinations, producing task-specific predictions through shared or independent heads. The framework is evaluated on synthetic data and real-world psychotherapy notes, demonstrating superior performance over fixed multitask and single-task baselines. This approach has significant implications for personalized healthcare, potentially improving mental health outcomes and treatment precision.", "questions": "Real-World Applicability:\nThe reliance on synthetic data for initial validation is a notable aspect of the framework. How do you envision the model performing in entirely real-world scenarios where data distributions may be more complex and nuanced? It would be useful to discuss potential strategies for adapting the framework to such environments and mitigating any synthetic bias.\nInterpretability and Clinical Workflow Integration:\nWhile the paper mentions the interpretability of the routing decisions, could you elaborate on how these insights could be practically integrated into clinical workflows? Discussing specific use cases or examples where the model's decisions could enhance clinical decision-making would strengthen its applicability.\nBroader Applicability:\nThe framework is primarily validated in the context of mental health prediction. How do you anticipate it performing in other domains with multimodal and multitask challenges? Exploring potential extensions or adaptations for broader applications would enhance the paper's significance.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality:\n\nStrengths:\nThe paper presents a robust and well-structured framework, with comprehensive experiments that validate the model's effectiveness in both synthetic and real-world settings.\nThe use of probabilistic routing to manage sample-specific heterogeneity is a strong methodological contribution, allowing for nuanced handling of data variability and task interdependencies.\nThe results demonstrate clear performance improvements over baseline models, showcasing the framework's practical benefits in terms of accuracy and interpretability.\n\nWeaknesses:\nWhile the framework is well-designed, the reliance on synthetic data for initial validation might not fully capture the complexities encountered in entirely real-world datasets.\nThe implementation details, particularly regarding the neural network architectures used for routing and expert selection, could be more thoroughly explained to enhance reproducibility.\n\nClarity:\nStrengths:\nThe paper is generally well-organized, providing clear explanations of the adaptive routing framework and the underlying concepts.\nVisual aids such as figures and tables effectively illustrate the model architecture, routing behavior, and experimental results, aiding reader comprehension.\n\nWeaknesses:\nSome technical aspects, such as the specific algorithms used for training the routing mechanism and expert modules, could benefit from additional detail to assist readers in understanding the implementation.\nThe integration of related work could be more cohesive, providing clearer context for how the proposed method differs from or improves upon existing approaches.\n\nSignificance:\n\nStrengths:\nThe paper addresses a critical challenge in multimodal multitask learning, offering a novel solution that could significantly enhance predictive performance in settings with heterogeneous data and complex task relationships.\nThe framework's potential applications in personalized healthcare and mental health prediction are particularly impactful, promising improvements in clinical decision-making and treatment outcomes.\n\nWeaknesses:\nThe broader applicability of the framework in other domains beyond mental health is not fully explored, which could limit its perceived significance in a wider context.\nThe potential impact on clinical workflows and how the model's interpretability could be leveraged in practice is mentioned but not deeply analyzed.\n\nOriginality:\n\nStrengths:\nThe introduction of a probabilistic routing mechanism that dynamically selects modality and task configurations on a per-sample basis is a novel contribution, setting this framework apart from existing multimodal and multitask learning approaches.\nThe integration of adaptive routing with heteroscedastic loss further enhances the model's originality, addressing both data variability and uncertainty in predictions.\n\nWeaknesses:\nWhile the approach is innovative, it builds on established concepts like mixture-of-experts and adaptive computation, which might moderate perceptions of originality.\nThe focus on specific clinical applications, while significant, may narrow the scope of originality if not extended to other potential domains.", "was_revised": true, "final_justification": "I have checked out author’s responses and the discussion, my concerns are addressed and i increase score to 5", "rebuttal": "### Real-World Applicability and reliance on synthetic data:\nWe thank the reviewer for their incisive comments. We agree that understanding model performance on genuine clinical data is critical for assessing real-world applicability. Our use of synthetic data is not a workaround but a deliberate choice to address a well-documented barrier in mental health AI. Psychotherapy datasets are notoriously difficult to access due to privacy constraints and institutional restrictions. Rather than sidestepping this challenge by switching to unrelated benchmarks, we chose to stay grounded in this clinically meaningful but under-resourced setting. We adopted empirical synthetic augmentation, a principled and increasingly recognized practice in the AI community.\n\nIn psychotherapy, structured numerical inputs (e.g., PHQ, GAD) are not simple metadata. They are semantically dense and clinician-generated symptom summaries. Treating these as purely numerical is limiting. We demonstrated an alternative: bidirectional transformation between structured and unstructured formats. Instead of reducing meaning to scalar scores, we textualized structured assessments and allow LLMs to integrate them alongside unstructured text inputs. This sets our approach apart and provides interpretability to clinicians, as the model now processes \"interpretable natural language reflections\" of clinical structure that traditional fusion pipelines and most MoE models overlook. While alternative datasets could have been used, our goal was to develop methods that address the real-world complexity of psychotherapy modeling. To assess generalizability, we evaluated the model on other real-world datasets, where it performed well, demonstrating broader applicability.\n\n\n### Interpretability and Clinical Workflow Integration:\nWe appreciate the attention given to interpretability and clinical integration, which are core motivations behind our routing framework.\n\nWhile Table 1 in the main manuscript summarizes the eight expert routes and their alignment with data configurations, the routing mechanism provides deeper insights into patient-level heterogeneity, which is a central theme of our framework. Specifically, we find that routing decisions reflect variability across patients in both modality informativeness and task dependency.\n\nFor example, the choice between single-task (STL) and multi-task (MTL) paths depends on the correlation between outcomes. Some patients exhibit strongly linked PHQ and GAD trajectories, for whom joint modeling via MTL is beneficial. Others show weaker or asymmetric relationships (e.g., rising anxiety without concurrent depression), prompting STL selection. This phenomenon, inter-patient variability in anxiety-depression comorbidity, is well-established in the clinical literature and emerges naturally through our routing decisions.\n\nLikewise, the selection among the four modality paradigms (T1: text-only, T2: textualized numerical + text, N1: numerical-only, N2: numerical + embedded text) reveals which input types are most predictive per patient. In some cases, therapist notes offer a richer signal for anxiety (T1 or T2), while structured scores are more informative for depression (N1 or N2), or vice versa. This variation cannot be captured through fixed modality fusion or conventional feature selection, which assume stable informativeness across patients.\n\nImportantly, routing also indicates when early modality fusion (T2/N2) is helpful versus when unimodal paths (T1/N1) suffice. For some patients, cross-modal signals carry complementary information; for others, one modality may lack relevant content, not due to quality issues, but because certain symptoms were not documented in that channel. This distinction is subtle but clinically meaningful. We will make these implications more explicit in the final version.\n\n### Broader Applicability\nTo evaluate the generality of our method, we applied the routing framework to three diverse datasets: **Yelp Reviews** (business feedback, 5,000 samples), **RateMyProfessor** (academic evaluations, 1000 samples), and **Customer Satisfaction** (structured service surveys, 2000 samples).  All contain both structured numeric inputs (e.g., ratings, survey scores) and unstructured text (e.g., freeform comments), aligning with the multimodal multitask structure of our method. Below we present comparative results for three key settings: **Soft routing (full model)**, **hard routing**, and a **non-routing ensemble**.\n\n| Dataset            | Config         | R² (Task 1) | R² (Task 2) | Δ vs. Full Model     |\n|--------------------|----------------|-------|-------|----------------------|\n| **Yelp**           | Soft Routing   | 0.2406 | 0.2263 | —                   |\n|                    | Hard Routing   | 0.2350 | 0.2200 | –0.0056 (–2.3%)     |\n|                    | No Routing     | 0.1790 | 0.1651 | –0.0616 (–25.6%)    |\n| **RateMyProf**     | Soft Routing   | 0.2850 | 0.2720 | —                   |\n|                    | Hard Routing   | 0.2800 | 0.2670 | –0.0050 (–1.8%)     |\n|                    | No Routing     | 0.2200 | 0.2100 | –0.0650 (–22.8%)    |\n| **Customer Sat.**  | Soft Routing   | 0.3200 | 0.3050 | —                   |\n|                    | Hard Routing   | 0.3150 | 0.3000 | –0.0050 (–1.6%)     |\n|                    | No Routing     | 0.2500 | 0.2400 | –0.0700 (–21.8%)    |\n\n\nThe full model with soft routing across all paths achieves the best performance across datasets. Hard routing performs slightly worse, with an average drop of 0.5 to 2.0 percentage points in $R^2$. Ablation studies show that the text-based paths (T1 and T2) are the most critical, while N2 adds useful complementary information. \n\n\n\n### Implementation details and specific algorithms used for training the routing mechanism\nThank you for requesting additional implementation details. We will include a detailed technical appendix in the revised supplementary material to ensure full transparency and reproducibility for each experiment. Training setup and algorithmic structure such as\n\nTraining parameters.\nOptimizer: Adam (lr=1e-3, weight_decay=1e-4),\n Gradient Clipping: max_norm=0.5,\n Learning Rate Scheduling: ReduceLROnPlateau (factor=0.7, patience=10),\n Early Stopping: patience=15 epochs,\n Batch Size: 8-16 samples,\n Training Epochs: 50-90 (convergence-dependent),\n\nRegularization techniques.  Entropy Regularization: Prevents expert collapse by encouraging diverse routing,  Weight Decay: L2 regularization on all parameters, \nand Gradient Clipping: Prevents exploding gradients\n\nInference algorithm. Soft Routing (Training): Uses continuous probability distributions, Enables gradient flow through routing decisions, Maintains exploration of different pathways and Hard Routing (Inference):\n Uses Gumbel-Softmax with temperature annealing,\n Discretizes routing decisions for interpretability,\n Formula: `hard_route = gumbel_softmax(log_probs, tau=0.1, hard=True)`\n\n### Integration of more related work\nWe thank the reviewer for this helpful observation. While our related work section was organized by topic, we agree that a more cohesive narrative would strengthen the framing of our contributions. In the revised version, we will clarify how our method unifies and extends several strands of prior work. Unlike earlier multimodal models that rely on fixed fusion schemas, our approach performs instance-wise adaptive fusion. In multitask learning, we move beyond fixed or statically gated architectures by routing tasks through dynamic expert paths conditioned on sample-level signals. Unlike prior routing or Mixture-of-Experts (MoE) architectures that operate at the task or modality level independently, our model performs joint routing over both axes. To reflect recent progress in adaptive computation, we also plan to incorporate several new references that complement our contributions. These include recent advances in sparse expert routing [1,2], dynamic gating in multimodal settings [3,4], adaptive MoEs for vision and language models [5,6], task-level expert allocation [7,8], and unified frameworks for scalable mixture-based inference [9,10]. We appreciate the reviewer’s suggestion and will revise the section to better integrate and position our approach within these developments.\n\nAdditional References to be Incorporated:\n\n[1] Zhou et al. (2022). Mixture-of-Experts with Expert Choice Routing. NeurIPS.\n\n[2] Riquelme et al. (2021). Scaling Vision with Sparse Mixture-of-Experts. NeurIPS.\n\n[3] Jain et al. (2023). DAMEX: Dataset-aware Mixture-of-Experts for Visual Understanding of Mixture-of-Datasets. NeurIPS.\n\n[4] Park et al. (2025). Resilient Sensor Fusion Under Adverse Sensor Failures via Multi-Modal Expert Fusion. CVPR.\n\n[5] Fedus et al. (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR.\n\n[6] Lepikhin et al. (2021). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. ICLR.\n\n[7] Huang et al. (2024). Harder Task Needs More Experts: Dynamic Routing in MoE Models. ACL.\n\n[8] Dai et al. (2022). StableMoE: Stable Routing Strategy for Mixture of Experts. ACL.\n\n[9] Kudugunta et al. (2021). Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference. EMNLP Findings.\n\n[10] Yu et al. (2024). MMOE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts. EMNLP."}, {"review_id": "YZjoH4vxEw", "reviewer": "Reviewer_o6nv", "summary": "This paper proposes a unified framework for per-sample adaptive routing in multimodal, multitask prediction, motivated by applications in psychotherapy where both structured assessments (e.g. questionnaires) and unstructured clinician notes coexist. The key idea is to dynamically route each input through different modality transformations (e.g. text-only, numerical-only, or fused representations) and task architectures (single-task vs. multitask), depending on input-specific signals. The model employs a hierarchical mixture-of-experts architecture with learnable routing networks that select both the optimal modality pathways and the degree of task sharing for each sample.", "questions": "- Soft routing requires computing outputs for all 8 expert paths per sample. Did you measure the inference-time computational cost of your method compared to fixed-path baselines? How significant is the overhead, especially for larger datasets?\n\n- You mention using entropy regularization to prevent expert collapse. Could you quantify how much this regularization affects routing diversity in practice? Without it, did you observe any experts dominating the routing probability mass?\n\n- Your results show that the textualized numeric + text path (T2) outperforms numerical + embedded text (N2). Could you elaborate on why textualization yields better performance? Do you believe this is due to language models capturing richer context, or could it reflect overfitting to the textual modality?\n\n- Do you foresee risks or limitations in transforming structured numeric data into textual form? For example, might subtle quantitative nuances be lost or misrepresented during textualization, especially for clinical scales?\n\n- Beyond aggregate routing probabilities, have you explored visualizing routing decisions at the individual patient level? Could such visualizations aid clinicians in understanding why the model made certain predictions?", "limitations": "Limitations have been discussed in section 5.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- **Novelty in Joint Adaptive Routing** The paper introduces a unified adaptive routing framework that jointly routes over both modalities and task structures (STL vs. MTL) on a per-sample basis. This is a meaningful extension beyond typical MoE architectures, which often focus solely on routing over experts for a fixed modality or task.\n- **Bidirectional Modality Transformations** The bidirectional transformations between structured and unstructured data (numerical → text and text → numerical) are a creative and flexible mechanism. They enable four distinct fusion paradigms (T1, N1, T2, N2) that adapt to missing or weak modalities, addressing a genuine practical challenge in clinical data.\n- **Strong Clinical Motivation and Application** The motivation is rooted in a real-world clinical problem and the paper is highly relevant for personalized medicine\n- **Uncertainty-Aware Modeling** The paper combines adaptive routing with a heteroscedastic loss, modeling prediction uncertainty.\n\n**Weaknesses**\n- **Reliance on Synthetic and Hybrid Data** The real-world experiments partially rely on a hybrid dataset that augments genuine psychotherapy records with synthetic encounters. This introduces potential domain shift, making it unclear how the model would perform on entirely real, diverse clinical datasets.\n- **Limited Task Diversity** The experimental validation focuses on only two clinical outcomes (PHQ-9, GAD-7). While well-motivated, this leaves open the question of whether the method generalizes to more diverse mental health outcomes and other clinical domains\n- **Computational Scalability Not Analyzed** The model considers eight expert paths (4 modality paths × 2 task configurations). However, there's no explicit analysis of computational cost. How does routing scale with increasing numbers of modalities, tasks, or larger datasets? What is the inference-time cost for large-scale deployment?\n- **Potential Overhead from Soft Routing** The soft routing approach computes a full mixture over all expert paths, requiring all experts to run per sample. The paper does not quantify the runtime penalty of soft vs. hard routing, nor how this affects latency in practice.", "was_revised": true, "final_justification": "After discussion with the authors where they have addressed my key concerns, I will like to raise my score to 6.", "rebuttal": "### Reliance on Synthetic and Hybrid Data\nWe thank the reviewer for their time, careful reading, and clear understanding of our paper. We agree that understanding model performance on genuine clinical data is critical for assessing real-world applicability. Our use of synthetic data is not a workaround but a deliberate choice to address a well-documented barrier in mental health AI. Psychotherapy datasets are notoriously difficult to access due to privacy constraints and institutional restrictions. Rather than sidestepping this challenge by switching to unrelated benchmarks, we chose to stay grounded in this clinically meaningful but under-resourced setting. We adopted empirical synthetic augmentation, a principled and increasingly recognized practice in the AI community.\n\nIn psychotherapy, structured numerical inputs are not simple metadata. They are semantically dense and clinician-generated symptom summaries. Treating these as purely numerical is limiting. We demonstrated an alternative: bidirectional transformation between structured and unstructured formats. Instead of reducing meaning to scalar scores, we textualized structured assessments and allow LLMs to integrate them alongside unstructured text inputs. This sets our approach apart and provides interpretability to clinicians, as the model now processes \"interpretable natural language reflections\" of clinical structure, that traditional fusion pipelines and most MoE models overlook. While alternative datasets could have been used, our goal was to develop methods that address the real-world complexity of psychotherapy modeling. To assess generalizability, we evaluated the model on other real-world datasets, where it performed well, demonstrating broader applicability.\n\n### Limited Task Diversity\nWe appreciate this thoughtful question. While we focus on two highly correlated but clinically distinct outcomes (PHQ-9 and GAD-7), our architecture was explicitly designed to generalize across larger sets of tasks.\n\nLet $t$ be the number of outcome tasks. We consider three task-structuring strategies:\n\n1. **STL + single MTL:** Each modality path is paired with $t$ STL heads and one MTL head covering all tasks. The total number of strategies is $(t + 1)$.\n\n2. **STL + pairwise MTL:** Instead of one MTL model, we train a separate MTL model for each **task pair**. The total number of strategies becomes $ \\left(t + \\frac{t(t - 1)}{2}\\right) =  \\frac{t(t + 1)}{2}$, which grows quadratically with the number of tasks.\n\n3. **STL + grouped MTL (recommended for scalability):** Tasks are grouped into $g$ semantically **related cluster of tasks**, each modeled by a shared encoder with group-specific heads. The total number of experts becomes $(t + g)$, where $g \\ll \\frac{t(t - 1)}{2}$. This strikes a balance between parameter efficiency and the ability to model inter-task relationships.\n\n\n\n\n### Q1: Routing overhead\nOur model incorporates a hierarchical routing system to dynamically assign each sample to one of eight expert pathways, defined by the Cartesian product of four modality configurations (T1, T2, N1, N2) and two task strategies (STL, MTL). We analyze the computational complexity introduced by this routing mechanism, relative to fixed single-path baselines.\n\nLet $P$ denote the computational cost of a single expert model (i.e., the cost of a forward and backward pass through one pathway), and let $R$ denote the cost of computing the routing distributions using the modality and task routers. The routers are shallow MLPs whose cost is negligible relative to $P$, so $R \\ll P$.\n\n**Fixed Baseline (No Routing).** In the simplest configuration, a sample is passed through a fixed expert (e.g., modality T1 with STL). The total cost per sample is\n$\nC_{\\text{fixed}} = \\Theta(P).\n$\n\n**Soft Routing.** In soft routing, all eight experts are evaluated per sample. The model computes weighted predictions by combining each expert's output according to the product of modality and task routing probabilities. Thus, the total cost per sample is:\n$\nC_{\\text{soft}} = R + \\sum_{i=1}^{4} \\sum_{j=1}^{2} P^{(i,j)}\n$\n\nAssuming each expert has similar cost $P$, we have:\n$\nC_{\\text{soft}} = R + 8P = \\Theta(8P + R) \\approx \\Theta(8P)\n$\n\n**Hard Routing.** In hard routing, a sample is assigned to a single expert using a differentiable approximation of the argmax operation (e.g., via Gumbel-softmax). Only one expert is evaluated, resulting in:\n\n$$\nC_{\\text{hard}} = R + P = \\Theta(P + R) \\approx \\Theta(P)\n$$\n\n**Relative Overhead.** Compared to a fixed-path baseline, soft routing incurs an $\\sim 8\\times$ increase in per-sample computational cost, while hard routing maintains nearly equivalent cost with a marginal increase due to the routing networks. in summary:\n\n| Routing Strategy           | Experts Evaluated | Cost per Sample      | Relative to Fixed |\n|---------------------------|-------------------|----------------------|-------------------|\n| Fixed Path (e.g., T1+STL) | 1                 | $\\Theta(P)$          | 1                 |\n| Hard Routing              | 1                 | $\\Theta(P + R)$      | $\\approx 1\\times$ |\n| Soft Routing              | 8                 | $\\Theta(8P + R)$     | $\\approx 8\\times$ |\n\n\n\n\n### Q2:  Entropy regularization\n\nWe conducted a detailed study to quantify the impact of entropy regularization on the Yelp Reviews dataset (5,000 samples). Without entropy regularization, we observed clear expert collapse. Adding entropy regularization substantially improved routing diversity.\n\n\n| **λ (Reg. Strength)** | **Routing Entropy (bits)** | **Expert Utilization**        | **Expert Collapse**     | **R² Performance**   |\n|------------------------|-----------------------------|-------------------------------|--------------------------|-----------------------|\n| 0.00 (None)            | 1.76                        | T1: 42%, others < 25%         | Yes (T1 dominates)       | 0.2350 (baseline)     |\n| 0.05 (Medium)          | 2.81                        | All experts ~22–28%           | No                       | 0.2406 (+0.6%)        |\n| 0.10 (Strong)          | 3.03                        | Uniform (~25% each)           | No                       | 0.2390 (stable)       |\n\n### Q3: Why does T2 (textualized numeric + text) outperform N2 (numerical + embedded text)?\nWe appreciate this thoughtful question. The key reason T2 performs better than N2 is the **semantic richness of the structured inputs** used in psychotherapy. These aren't generic numeric features or metadata; they are clinician-generated answers to carefully constructed diagnostic questions, where each one probing nuanced aspects of emotional and behavioral states. For example:\n\n> *\"Over the past two weeks, how often have you felt nervous, anxious, or on edge?\"*\n\nThe numerical response (e.g., 0–3) is a highly compressed signal. Treating it as a scalar feature (as done in N2) loses the context in which it was obtained. In contrast, our textualization step in T2 reverses this abstraction by expanding numeric inputs into natural language summaries. This allows pretrained language models to process them **not as isolated numbers**, but as part of the broader **semantic narrative** of the patient’s condition.\n\nImportantly, in T2, these textualized summaries are **concatenated with the therapist’s notes**, which are themselves rich in semantic and clinical information. The resulting input is processed through a transformer-based language model that jointly embeds this hybrid text. This **early fusion** in the **language space** enables the model to learn **cross-correlations between the structured clinical assessments and the therapist observations**, a pattern that cannot be easily captured when embedding text into fixed-length vectors and appending it to a numerical input, as in N2.\n\n### Q4: Do you foresee risks or limitations in transforming structured numeric data into textual form?\nTextualizing structured numeric data can introduce semantic richness and interpretability, especially in domains like psychotherapy, where numeric scores often summarize clinician-assessed symptoms. In our setting, this approach proved beneficial. However, this benefit is not universal. In domains or tasks where raw numeric precision is more informative, or where numeric variables lack semantic grounding (e.g., lab measurements or sensor data), textualization might introduce unnecessary abstraction or noise. Important quantitative subtleties could be diluted when mapped into language space.\n\nThat’s precisely why our architecture includes all four modality paths. The utility of textualization is **input-, task-, and domain-dependent**, and there’s no universally optimal transformation. Our probabilistic routing mechanism addresses this by dynamically selecting the best-suited path for each sample. If textualization helps, the model routes toward T2; if not, it can favor N1 or N2 instead. This flexibility allows the model to learn when to rely on textualization and when to bypass it, rather than assuming a fixed modality fusion strategy.\n\n### Q5: Visualizing routing decisions at the individual patient level\nYes, understanding sample-level routing is important for interpretability, especially in clinical settings where model transparency can support decision-making.\n\nWe address this in Figure 2 (Middle), which presents a **clustered heatmap of routing probabilities over the four modality paths (T1, N1, T2, N2)** for individual samples. This visualization shows that the model **segments the input space into distinct routing patterns** and assigns different modality preferences based on the characteristics of each patient’s input. These clusters align with latent structure in the data and help explain why certain modality paths are favored.\n\nWhile this figure presents a clustered view across samples, it **implicitly reveals per-sample routing decisions**, and similar visualizations can be generated for individual cases."}, {"review_id": "QLx9dQbPu6", "reviewer": "Reviewer_8T9F", "summary": "The paper proposes a unified framework for per-sample adaptive routing in multimodal, multitask prediction, focusing on mental health applications. It dynamically selects optimal data pathways (structured, unstructured, or fused) and task-sharing strategies (single-task or multitask) for each input using a probabilistic gating mechanism. Evaluated on synthetic data and real psychotherapy records, the model outperforms fixed baselines in predicting depression and anxiety outcomes.", "questions": "**Questions and Suggestions**\n\n1. **Ablation Study Design**\n   Could the authors clarify how the ablation studies were conducted, if any? Specifically:\n\n   * Were parameter counts controlled across ablated variants?\n   * Were all models (including fixed-path baselines) trained with the same set of hyperparameters?\n   * If not, how do the authors ensure that performance improvements are due to the adaptive routing mechanism rather than model capacity or training discrepancies?\n     A detailed, fair ablation setup is essential to isolate the contribution of each component. A clear answer here would improve confidence in the reported gains.\n\n2. **Clarity on MoE Design and Expert Modules**\n   The paper claims to adopt a Mixture-of-Experts (MoE) framework, but the actual architectural design seems limited to gating over modality/task paths—without clearly defined expert subnetworks. Could the authors:\n\n   * Clarify what constitutes an \"expert\" in their setup?\n   * Specify whether each expert has independent parameters, or whether the same backbone is reused with only gating on inputs/outputs?\n     As it stands, the model appears closer to a soft ensemble over different input transformations than a true MoE. A detailed architectural diagram or parameter-sharing explanation would help clarify this point. If experts are not truly independent, the originality and significance of the method may need to be reconsidered.\n3. **Baseline Selection and Use of Recent Models**\n   Have the authors evaluated their approach against stronger or more recent baselines, such as:\n\n   * Standard BERT-based encoders with task-specific heads?\n   * Multimodal fusion models using pretrained language models with structured data adapters (e.g., GatorTron-style or Med-BERT variants)?\n     Given the prevalence of BERT-like models in clinical NLP, it is important to understand how the proposed method compares to these widely used, stronger baselines. Including such comparisons would better contextualize the reported performance gains and strengthen the case for the proposed approach.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. **Flexible and Unified Framework**: The paper introduces a modular architecture capable of adaptively routing inputs across multiple modality combinations and task-sharing strategies. This per-sample adaptivity is well-motivated in clinical prediction tasks with heterogeneous data.\n\n2. **Interpretable Routing Decisions**: The probabilistic routing mechanism yields interpretable outputs that align with clinical expectations, offering potential benefits for real-world deployment in healthcare settings.\n\n---\n\n**Weaknesses:**\n\n1. **Limited Real-World Validation**: A substantial portion of the experiments relies on synthetic data, which may not accurately reflect real-world complexities or generalize well to clinical applications. The use of hybrid synthetic-real data further undermines confidence in the method's effectiveness in practical settings.\n\n2. **Lack of Ablation Studies**: The paper does not provide sufficient ablation experiment details to support the contribution of each component (e.g., routing, heteroscedastic loss, modality transformations). This limits the reader’s ability to assess what aspects of the model drive performance improvements.\n\n3. **Weak Baseline Comparisons**: The baselines are limited and insufficiently explored. The claimed gains over simple fixed-path models may result more from soft averaging than meaningful architectural innovation.\n\n4. **Low Originality**: The core idea—softmax-based selection over modality-task paths—is a straightforward extension of the feature selection and shared decoder technique, and I don't see why it is called MoE. The conceptual novelty is modest, and the methodological contribution may not be significant enough for a top-tier venue like NeurIPS.", "was_revised": true, "final_justification": "Part of my questions on MoE details and baselines have been resolved. Still the ablation exp models having huge difference on parameter size, which I still believe is a potential risk factor. Thus I will update my rating to reflect the updates.", "rebuttal": "### Limited Real-World Validation\nWe thank the reviewer for their comments. We agree that understanding model performance on genuine clinical data is critical for assessing real-world applicability. Our use of synthetic data is not a workaround but a deliberate choice to address a well-documented barrier in mental health AI. Psychotherapy datasets are notoriously difficult to access due to privacy constraints and institutional restrictions. Rather than sidestepping this challenge by switching to unrelated benchmarks, we chose to stay grounded in this clinically meaningful but under-resourced setting. We adopted empirical synthetic augmentation, a principled and increasingly recognized practice in the AI community. While alternative datasets could have been used, our goal was to develop methods that address the real-world complexity of psychotherapy modeling. In order to address the reviewer's concern about data and ablation studies, we evaluated the model on another real-world dataset presented below. \n\n### Q2: Clarity on MoE Design and Expert Modules\nWe appreciate the reviewer's attention to the architecture and welcome the opportunity to clarify. Our model does implement a Mixture-of-Experts (MoE) architecture, though with a design tailored to the structure of multimodal and multitask learning in clinical data. \n\nIn our architecture, **each expert corresponds to a distinct (modality, task paradigm) route**, such as N1-STL or T2-MTL. These experts are modular, **parameterized independently**, and trained jointly with the router. Each expert contains its own encoder and decoder, **not just a shared backbone with input gating**. For example: T2 expert use a text transformer (BERT) on textualized inputs, N2 experts use MLPs on structured embeddings, STL experts include two independent heads, and MTL experts share a trunk with separate task heads. **No weight sharing occurs between experts**, consistent with standard MoE literature. Each expert defines a full forward pass pipeline.\n\nA two-stage **probabilistic routing network** computes **per-sample** distributions over (1) modality paths and (2) task paradigms. These define a mixture distribution over the 8 experts and the final loss aggregates **heteroscedastic uncertainty-aware loss terms** across all expert paths. This makes routing **semantically and mathematically non-trivial**. Unlike feature selection or soft gating over inputs, the router **selects complete computational paths**, not just features, and unlike ensemble methods, routing is trained jointly with expert parameters via backpropagation, not post hoc weighting.\n\n### Q1: Ablation Study Design\nWe conducted a comprehensive ablation study on the Yelp Reviews dataset, using 2,000 samples with multi-modal inputs: Numeric questionnaire features, BERT-based text embeddings (384 dimensions), and BERT-based embeddings for textualized numeric features.\nThe experiments involve parameter-matched variants that isolate the effect of different routing paths and fusion mechanisms. The two prediction targets are star rating and sentiment. All models are trained with unified hyperparameters and a fixed 80/20 stratified train-test split.\n\nThe model variants and their purposes are listed below:\n\n| Model | Architecture | Parameters | Purpose |\n|-------|-------------|------------|---------|\n| **Full_Routing** | Complete routing model | 128,381 | Baseline |\n| **N1_Only** | Numeric features only | 2,530 | Ablate text features |\n| **N2_Only** | Numeric + Text features | 27,106 | Ablate textualization |\n| **T1_Only** | Text features only | 26,786 | Ablate numeric features |\n| **T2_Only** | Text + Textualized features | 51,362 | Ablate numeric features |\n| **No_Routing** | Same architecture, no routing | 51,682 | Ablate routing mechanism |\n| **MLP_Baseline** | Standard MLP | 109,474 | Traditional neural baseline |\n\nAll ablation variants except Full_Routing are parameter-matched (~15k parameters ±20%) to fairly compare architectural changes without capacity bias.\n\nThe ablation results are summarized below:\n\n| Model | MSE_T1 | MSE_T2 | \n|-------|--------|--------|\n| **Full_Routing** | 0.1042 | 0.0835 | \n| N1_Only | 0.1203 | 0.0817 \n| N2_Only | 0.1131 | 0.0816 \n| T1_Only | 0.1083 | 0.0818 \n| T2_Only | 0.1093 | 0.0821 \n| No_Routing | 0.1123 | 0.0818 |\n| MLP_Baseline | 0.1208 | 0.0818 | \n\n\n\n### Q3: Baseline Selection\nThank you for your valuable feedback. We initially experimented with clinical-BERT and several other text encoding variants; however, these changes did not yield significant improvements in our tasks. The primary gains in our work come from the multitask learning setup, the textualization of numeric features, and, most importantly, our adaptive routing mechanism for modality fusion. Our main contribution lies not in proposing new text encoders, but in designing an effective approach to fuse heterogeneous modalities and routing tasks, which we believe addresses a critical challenge in multi-modal learning. However,\n\nWe conducted a comprehensive comparison with state-of-the-art BERT-based baselines on the Yelp dataset:\n\n- **Dataset**: 2,000 Yelp review samples with structured numerical features and BERT embeddings (384D)\n- **BERT Model**: all-MiniLM-L6-v2 (Sentence Transformers)\n- **Tasks**: Two regression targets (rating prediction and sentiment analysis)\n- **Training**: 100 epochs with early stopping, Adam optimizer, MSE loss\n\n## Complete Model Comparison Results\n\n| Model | MSE_T1 | MSE_T2 | \n|-------|--------|--------|\n| **BERT Task-Specific Head** | 0.2080 | 0.2019 | \n| **Multimodal BERT Fusion** | 0.2103 | 0.2026 | \n| **Med-BERT Variant** | 0.2078 | 0.2022 | \n| **GatorTron-Style** | 0.2099 | 0.2034 | \n| **Our Routing Model** | 0.2058 | 0.2019 |"}]

Author rebuttal:

Meta-review: The paper designs a framework for mixture of multimodal experts for health applications with multimodal data and multiple tasks. The framework contains novel aspects compared with previous works including modality-task experts and bidirectional modality transformations. Routing in training is done via soft routing, which improves the performance but increases the cost by 8x. The reviewers are also concerned with the limited scope of the experiments. The rebuttals include additional experiments on more datasets and partially address this concern.

Overall, the majority of the reviewers support the paper but one reviewer places it slightly below the bar. Given that one reviewer strongly supports the paper, on balance, the paper seems above the bar.
Decision: Accept (poster)