MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Series
Authors: Payal Mohapatra
Yueyuan Sui
Akash Pandey
Stephen Xia
Qi Zhu
Venue: NeurIPS 2025 spotlight ()
Keywords: Multimodal Time series
Sparse Attention
Primary area: deep_learning
TL;DR: Multivariate processing of diverse sensing modalities is limiting; we propose an adaptive framework to conduct explicit intra- and cross-modal learning using sparse attention that can handle arbitrary modality-missingness adeptly.

Abstract: From clinical healthcare to daily living, continuous sensor monitoring across multiple modalities has shown great promise for real-world intelligent decision-making but also faces various challenges. In this work, we argue for modeling such heterogeneous data sources under the multimodal paradigm and introduce a new framework, MAESTRO. We introduce MAESTRO, a novel framework that overcomes key limitations of existing multimodal learning approaches: (1) reliance on a single primary modality for alignment, (2) pairwise modeling of modalities, and (3) assumption of complete modality observations. These limitations hinder the applicability of these approaches in real-world multimodal time-series settings, where primary modality priors are often unclear, the number of modalities can be large (making pairwise modeling impractical), and sensor failures often result in arbitrary missing observations. At its core, MAESTRO facilitates dynamic intra- and cross-modal interactions based on task relevance, and leverages symbolic tokenization and adaptive attention budgeting to construct long multimodal sequences, which are processed via sparse cross-modal attention. The resulting cross-modal tokens are routed through a sparse Mixture-of-Experts (MoE) mechanism, enabling black-box specialization under varying modality combinations. We evaluate MAESTRO against 10 baselines on four diverse datasets spanning three applications, and observe average relative improvements of 4% and 8% over the best existing multimodal and multivariate approaches, respectively, under complete observations. Under partial observations—with up to 40% of missing modalities—MAESTRO achieves an average 9% improvement. Further analysis also demonstrates the robustness and efficiency of MAESTRO's sparse, modality-aware design for learning from dynamic time series.

MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Series
Payal Mohapatra
Yueyuan Sui
Akash Pandey Northwestern University
Stephen Xia
Qi Zhu
Evanston, IL, USA
(payal.mohapatae mohapatra, qzhu) qchu
Abstract
From clinical healthcare to daily living, continuous sensor monitoring across multi- ple modalities has shown great promise for real-world intelligent decision-making but also faces various challenges. In this work, we introduce MAESTRO, a novel framework that overcomes key limitations of existing multimodal learning ap- proaches: (1) reliance on a single primary modality for alignment, (2) pairwise modeling of modalities, and (3) assumption of complete modality observations. These limitations hinder the upplicability of these approaches in real-world multi- modal time-series settings, where primary modality priors are often unclear, the number of modalities can be large (making pairwise modeling impractical), and sensor failures often result in arbitrary missing observations. At core, MAESTRO facilitates dynamic intra- and cross-modal interactions based on task relevance, and leverages symbolic tokenization and adaptive attention budgeting to construct long multimodal sequences, which are processed via sparse cross-modal attention. The resulting cross-modal tokens are routed through a sparse Mixxture-of-Experrt (MoE) mechanism, enabling black-box specialization under varying modality com- binations. We evaluate MAESTRO against 10 baselines on four diverse datasets spanning three applications, and observe average relative improvements of 4% and 8% over the best existing multimodal and multivariate approaches, respectively, un- der complete observations. Under partial observations- with up to 40% of missing modalities Moddlittis-MMASTRRR achieves an average 9% improvement. Further analysis also demonstrates the robustness and efficiency of MAESTRO's s sparse, modality-awan design for learning from dynamic time series.
Introduction
Many real-world applications- as activity recognition [2], stress monitoring [46, 39], sleep- stage detection [13], and clinical decision-making [20]-have been significantly enhanced by the integration of wearable and inconspicuous continuous-sensing technologies. Devices such as smart- watches, smartphones, and chest straps collect time-series data that encode information about various physiological and behavioral phenomena (e.g., Electrodermal Activity (EDA) for skin conductance, accelerometers for motion, and Electrocardiogram (ECG) for heart activity). Given the common temporal structure of these signals, they are often abstracted as multivariate time-series data [37]. However, this simplified representation can be suboptimal due the inherent heterogeneity of sensing modalities and the varied task relevance of each modality and their interactions. An alternative is sensor fusion, but existing approaches [68, 57, 17] are often application- speecfif and heuristic. Even within single application such as estimating heart rate from multimodal sensing, studies may employ diverse strategies, ranging from early-fusion wavei-wavelength [35] and multi-site PPG [34] to late fusion with temperature [36], reflecting lack of consensus how to systematically handle modality interactions. Hence, there is a pressing need for a general framework that can automatically learn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

task-specific intra- and inter-modal dependencies without relying on ad-hoc heuristics or exhaustive fusion searches.
While significant strides have been made in the multimodal learning domain [25], most contemporary approaches that consider time-series binding learning across multiple modalities to single anchor modality [12, 44, 40J, which naturally risks over-reliance on a predefined primary modality; conducting explicit pairwise interaction mod- cling, which has shown continued promise from early efforts in cross-modal transformer, MULT [52], the more recent FlexMoE [67]. However, such modeling becomes combinatorially expensive as the number of modalities increases a common occurrence in rich, multi-sensor applications. Another important consideration in designing multimodal learning frameworks for time-series from real- world sensing applications is supporting learning from arbitrary combinations of sensing modalities, as sensor malfunction is commonplace [33, 47, 38].
of the modalities fall into of two categories-
Our Approach. To improve multi- modal learning for time-series data- while overcoming the limitations of ex- isting multimodal learning frameworks and handling samples with arbitrary sen- sor combinations we propose MAESTRO, which features per-Modal sparse atten- tion with an Adaptive attention bud- get, followed by Efficient Sparse cross-
0.50
mmw.
0.70
0.00
Multivaritee Transformer (MAESTRO)
Missingness 1%5
Exeriph Strees
Attenyy Sanoo Fumians
Albertonn
modal attention handle long multi-
Truditonal Muthorion
Propcoed HALSING The momee Mutimod
modal Temporal sequences, which are then Routed using a dynamic Mixture-of- Experts (MoE). Figure 1 presents a jux- taposition of our proposed approach and classical multivariate time-series process- ing in the context of a stress monitoring ap- plication, highlighting the arbitrary sensor missingness that our approach is designed to handle. Instead of projecting all interrelated observations each time step into common latent space and modeling their interactions using a classical self-attention mechanism under a multivariate setting, MAESTRO framework applies cross-modal attention within casalittss-fcilitaattii disen- tangled latent projections for each modality-ann explicitly models interactions across modalities at all time steps. This results in significant improvements in prediction accuracy (around 10% for the stress monitoring application; see Section 3 for detailed experimental results). More specifically, our contributions can be summarized as follows:
400
Figure 1: Illustration of traditional multivariate processing our multimodal handling of sensor data, highlighting our method's superior performance and robustness.
We present novel perspective modeling time-series data from multiple diverse sensing modalities for multimodal learning, enabling the discovery of task-relevant, modality-peeiifi and cross-modal interactions.
We propose MAESTRO, novel multimodal learning framework that integrates four key innovations: (1) symbolic tokenization with reserved tokens for missing data, (2) adaptive attention budgeting guided by modality availability and relevance, (3) sparse attention over long multimodal sequences to capture rich cross-modal context, and (4) loss-free, MoE-based dynamic routing that adapts to varying modality observations.
We conduct extensive experiments four real-world datasets (with 5 to 17 modalities) and demon- strate that MAESTRO consistently outperforms 10 multivariate and multimodal baselines state-of of the-art art MoE approaches- under diverse modality availability conditions, highlighting its robust performance and its potential to advance multimodal time-series learning.
2 Related Works
Multivariate and Fusion Approaches. Approaches such as InceptionTime [16], R [51], and Transformers [53, 10] have established themselves as leading models [37, 3] for multivariate time-series tasks. Beyond general multivariate modeling, recent research has demonstrated the benefits of variate-specific strategies-such double attention mechanisms [29, 62, 59] and channel

---

selection [63, 58, 43]-which -which enable more granular modeling of dependencies among variables. In application-specifee contexts, sensor fusion paradigms range across a spectrum, from early fu- sion [ multivariate processing is among the earliest forms- to late fusion methods [57, 17] like ensemble learning [5], which integrate predictions at the decision level. However, these fusion strategies are often heuristic and face challenges with generalization and robustness to missing data.
Multimodal Approach. While multimodal learning is predominant in audio-visual domains such as speech recognition [9], video understanding [9. 11. and affective computing [24], it is increas- ingly being applied to fields such as robotics [19, 22, 21], numan-compurt interaction [8], and healthcare [11, 65, 57], which require the integration of heterogeneous time-series data from diverse sensors. Recent efforts have acknowledged this heterogeneity and proposed -heooeii frameworks to formalize diverse multimodal interaction phenomena [54, 26, 4, 27]. Some works high- light the need for modeling both intra- and inter-modal interactions [7], while others propose explicit interaction modeling [66, 18] to capture emergent cross-modal knowledge that is task-relevant- further emphasizing the superiority of task-specific multimodal learning [70] over self-supervised multimodal representation learning [27]. Popular multimodal approaches include Tensor Fusion, multimodal Transformers, and multimodal MoEs [66, 61]. Multimodal works vary in their definition of someeit" framing it through data encoding mechanisms [14], and other times more qualitatively, based on how the signal is manifested or experienced [4]. As a result, in prior evaluations involving time-series data, multiple sensor streams are often simplified and treated as a single modality (see Table 2 in MultiBench [25]. a benchmarking resource). This treatment is suboptimal when sensors monitor fundamentally different aspects of the underlying phenomenon.
theoretic
Learning under Missing Modalities. One of the practical challenges of multimodal learning is the unreliability of all modalities during inference. A commonly adopted approach to handle missingness is the parameterized reconstruction of missing modalities [31, 32, 50], which can be computationally intensive with increasing number of modalities. Recent multimodal approaches address this issue through similarly motivated strategies, such as missingness bank completion within the MoE framework [67, 15, 61], or cross-modal mannfer-particuaarty when learning is achieved by binding to a primary modality [12, 40, 42] or leveraging shared modality representations [55].
3 Methodology
3.1
Preliminaries and Notations
Often time series data from distinct sources are abstracted as multivariate sequences in RxT our approach explicitly models time series from different sensors as multimodal data. We begin by formally defining multimodal time series:
Definition 3.1 (Multimodal Time-series). A multimodal time series consists of M 2 2 modality- specific time series that collectively describe underlying phenomenon through complementary, largely semantically-disjonn observations. For a sample E [I,...,N., we define the multimodal input as:
M
(A. MM TM
Modality
Modality M
where 2²[(t] E RD denotes the feature vector at time for modality j, with D features over T time steps. Each modality x! forms a multivariate time series in D The modality streams are assumed to be largely semantically-disjonn (as empirically validated later in Section 4.2), meaning that:
Afzk D'TT +R R x such that fzx(x)) ott Vj k.
Problem Statement. Given a multimodal time-series dataset D = {M, v1) i I,..., N}. denotes set of M modality-specifie time series with of D XT and V: E {1, (1...., C} is the class label, the objective is to learn a predictor f Xs + fl...., C}, where, o5 D2
where M; = "o.,...,,.""" ..MM
{ MM: Mi
xs
Code is available at https htte::///ttttee http:/shtt....oooooooooaat//tt/t
3

---

Prediction Head
Sparse MoE Layer
Sparss Attention
Cross- Cross-Modae Modal Learning with
Router
Learning Adaptive
Modility Modialty-Spechtt Specttic
Sparse Attention
Modality
Temporal Encoding
Adaptive Attention Budget
Sensing: Modality
Combinations
Symbolic Tokenization
Arbitrery
Sample
Sceple
Scll-Attenion
Modality
Actention
Natworks
Presencc
Figure 2: Overview of approach, MAESTRO. Input data from arbitrary combinations of sensing modalities is tokenized using symbolic approximation, where reserved symbol is used denote missing modalities. A learnable attention budget gate to allocates modality-wiss attention capacity for sparse-attentine attention -aastd modality- specific encoders. The resulting moddlitty-speciinn features are concatenated and combined with modality and positional embeddings, forming long multimodal sequence, which is processed by a sparse cross-modal multihead-attentee layer(s). The resulting tokens are routed through Sparse Mixture-of -f-ppers module, enabling dynamic specialization under arbitrary observability conditions. Finally, classifier maps the aggregated representation to task predictions.
for an arbitrary subset of available modalities S S (1,...,MM, M). which may vary per sample.
In this setting, not all modalities contribute equally: some are individually task-relevant, others are informative only through interactions (as characterized by Liang al. [26]), and some may introduce noise or cause negative transfer (as empirically demonstrated later in Section 4.2). The number of possible interactions grows combinatorially with M, making it impractical to explicitly model all subsets in rich multisensor scenarios, where typically M 2 4. Additionally, at inference time, the available modality set S, may vary arbitrarily due to real-world constraints. We refer to this setting-where where the combination of available sensing modalities changes across samples- as dymamic time-series this paper. The objective is to learn a predictor f that adapts to each S,, identifies and exploits informative modalities and interactions, suppresses spurious ones, and generalizes effectively across dynamic time-series.
3.2 Our Approach
In this section, we detail each component of MAESTRO, which is specifically designed to efficiently address key challenges in multimodal time-series learning, such as the presence of numerous sensing sources and abrupt sensor failures that lead to arbitrary combinations of available modalities, all while maintaining high performance.
We first reduce sequence length and encode missingness via symbolic representation. Next, an -budge gate, informed by modality relevance and availability, guides each modality-specific encoder. Their outputs are concatenated with modality and temporal position embeddings to form unified sequence, processed by a sparse eross-modal attention network to model task-relevant interac- tions. The resulting tokens are routed through Sparse MoE for final prediction. A comprehensive illustration of MAESTRO is provided in Figure 2, and the following sections present detailed design descriptions.

---

3.2.1 Missingnes--aware Symbolic Tokenization for Multimodal Time Series
To represent time-series data from each modality for efficient downstream processing, we leverage the symbolic aggregate approximation (SAX) method [28]. SAX is computationally efficient and preserves pairwise relational structure. Particularly in our MAESTRO framework, it gives opportunity to reserve symbol represent missingness. We first normalize the input time series ay, and then apply piecewise aggregation to convert x[t] (dropping indices i and j for brevity) into a compressed representation [[w]:
W
W
[w]
x[t].
(w
where 7' is the length of the original time series and W is the number of aggregated segments after compression.
To ensure an equiprobable distribution of symbols, we follow the empirical observation by Lin et al. [28] that normalized time series sequences follow Gaussian distribution. We partition this distribution into a equal-sized areas under the Gaussian curve, referred to as regions, using breakpoints (So, , 3. where each region corresponds to a unique symbol. The normal- ized piecewise aggregated value a {[w] is then mapped to a symbolic token s[vo] E (18,....88)] based the breakpoint interval in which it falls, denoted using the mapping function ((((w)).
To support arbitrary missing modalities, we reserve an additional symbol so indicate missingness:
s[w]
so, (+([w))
if modality is missing at window otherwise
Strke
Time
Figure 3: Recunstruction from sym- bolic tokenization of a PPG signal.
A visualization of this transformation is shown in Figure 3. Because symbolic tokenization is applied independently each modality, we can accommodate modality- specific sequence lengths and select compression parameters W individually. With shared alphabet size a across modalities and so indicating missing tokens, the symbolic representation of modality for sample is:
of E (so, S1, sa) D'xW5
where D is the number of variates in modality j. and W is the post-compression sequence length.
Theoretical Motivation: Symbolic Representation Preserves Multimodal Relational Structure. We build on the MINDIST result from Lin et al. [28]. which guarantees that the symbolic distance between time series (Distsym) lower-bounds their original Euclidean distance. We extend this the multimodal setting by assuming that all modalities are normalized prior to symbolic conversion, which yields the following corollary:
Corollary 3.2 Modaa Relational Preservation with Bounded SAX Distortion). Let modalities i and m produce SAX-based symbolic representations with shared alphabet size and compression lengths W, and Wm, respectively. Suppose the MINDIST nower-bound property holds within each modality:
(6,8%) 8.) S 1| 12,1,, Dist (5,,
Distim (S m < ||x:""
m 1|
and that the SAX approximation errors bounded:
| 21,12 Dist (5), $ Ej. Then, the difference in symbolic distances across modalities is bounded as:
|>=;- - 2F'l2 1|22 Dist (3 .8% S Cm
Disti (5, st) Dist (s." ( sK < | - 1
||=""
-
xx + + Em.
Implication. This result above shows that
symbolic representations not only preserve relational structure within each modality, but also maintain the relative contrast in sample similarity across modalities. Specifically, if two samples are more similar in modality j than in modality m. this relationship is preserved after symbolic discretization. This implies that symbolic tokenization retains meaningful cross modal structure. A proof sketch is provided in Appendix A.
5

---

3.2.2 Adaptive Sparse Attention for Intra-modal Learning
Following the tokenized representations described in Section 3.2.1, we extract task-specife intra- modal features using dedicated encoders for each modality. Based on the standard formulation of Vaswani et al. [53], we first apply sinusoidal positional embeddings (PExin to the modality stream s and compute the query, key, and value matrices Q' E RRd _K' RdxL V E RdxLvs. However, instead of the canonical self-attention mechanism, we employ a sparse self-attention strategy, (Q', V inspired by Zhou et al. [69], with key modification: the sparsity budget is adaptively controlled by the learned function a am,; 0a), where m, 1 logit vector indicating the presence of each modality for an input sample sf Details on training a a(m;; 0,) provided in Section 3.2.3. We denote the budget per modality omit indices and j in the following description of modality-specifff encoders with adaptive sparse attention.
[z] For brevity,
For each query q E Q, random subset of keys denoted as K' C K, is sampled where K'I log LK [69]. The dot product q K' is used compute the max-mean sparsity metrie Plq. K') proposed by Zhou et al. [69] given below:
qk Vd
qk'
P(q, K')
max
(1)
KEK'
K
Vd
KEK'
This max-mean measurement evaluates the query's attention diversity. The queries with higher P(g, K") scores contain more distinctive information. Next, the top-u queries-where v log La- wi-with the highest sparsity scores are selected to favor more diverse queries [69]. The top-2 queries are selected independently for each attention head to avoid excessive information loss due to sparsification. We then distil the self-attention outputs using ID convolutional layers and max- max-pooling followed by residual connections. The overall structure the modality-specinf encoders, g: (s, u) 2, is summarized below:
= + PE
(positional encoding)
QK
= AA V) Softmax
V
(sparse multi-heat attention: Q is sparse)
Vd
= 88 + 8
(residual connection)
distil($) + mmxxpool())
(attention distillation)
3.2.3 Modality-Aware Attention Budgeting
Our modality-specific encoders previously described are designed using a sparse attention mechanism. We propose adaptively learn this sparse attention budget for each modality based on its task relevance and availability by modulating the maximum attention budget, denoted by B E R+, resulting in budget vector u, E R as:
The attention budget u, parameterized by 6a, is computed
u; = aa (G,m, + <(1 - -m:); 0.)) -3]
where is small constant used for numerical stability, and (-(-) denotes the sigmoid activation. The gating function (m,;; 0a) adaptively allocates attention capacity based on both modality availability and task relevance, as it is trained with the task-specific objective.
3.2.4 Cross-Modal Learning using Sparse Attention for Long Multimodal Sequences
Let MM denote the modality-speciff embeddings obtained from the previously described en- coders. At this stage, our goal is devise an optimal learning strategy for inter-modal interactions that are relevant to the task while avoiding the effect of noisy or fask-irrelevant modalities that may degrade predictions (illustrated with case-study in Section 4.2).
Constructing the Multimodal Sequence. We concatenate features along the temporal dimension to form a unified representation. Following standard practice [64]. we first add modality-speciie and temporal positional embeddings to each Zj. denoted as ME(z,) and PE(z,), respectively: then concatenated along the temporal axis
2, z, + ME( + PE(z,). These representations to form unified multimodal sequence c = Concatiime (21, 22, 2ss), where M the number of modalities, and I DxL with L Σラ Lg-

---

To capture inter- modal dependencies, we apply sparse attention to the con- catenated sequence c. Let ć (Q., Ve), where As is the sparse multi-head attention operator, and Qe. Ke, and Ve are the query. key, and value vectors derived from This strategy offers the some key advantages: (1) it supports varying sequence lengths for different modalities. For example, in activity recognition tasks, accelerometers typically have a higher sampling rate (generally 25 Hz) than electrodermal activity sensors (generally 4 Hz). Uniformly resampling them can be suboptimal; (2) it facilitates time-ngrying cross-attention querie from one modality can attend to keys from the same modality as well as from other modalities. This acts as a generalized form of self-attention within a modality and cross-attention across modalities, enabling the learning of relevant inter-modal interactions; and (3) it is independent of explicit pairwise interaction modeling, which is impractical for modalities 2 4, as is typically the case in sensing applications. However, one of the key challenges with this approach is the increased sequence length, where Ż = E Ly, especially as the number of modalities grows. To address this, we propose leveraging the sparse-attentinn mechanism handle this long multimodal sequence.
Applying Sparse-Cros
Table I; Cross-attention methods' complex- ity. L: multimodal sequence length, M: number of modalities, Luar: longest modal- ity sequence length.
Method
Time
Space
The sparsity budget is controlled by a fixed parameter (similar to u = I, in the context of the modality-specífic pparse-attention encoders in Section 3.2.2). This sparse attention mechanism reduces the computational complex- ity from (i.²) to (ī. log i.). adeptly handling this long concatenated multimodal sequence. Table summarizes the space-time complexity of our proposed cross-modal sparse attention-based framework against its dense and pairwise cross-modal counterparts. After applying sparse attention, undergoes similar distil transformation and residual connections, described in Equation (2), yielding the final output e I f Figure 9 in Section 4.2 illustrates an instance of and confirms its utility in capturing cross-modal relations.
Dense Pairwise Sparse (Ours)
O(13) O(M 1
O(i2) O(M L
O log L)
Ol log L)
3.2.5 Sparse Mixture-of- Routing and Optimization
Inspired by the success of curriculum learning strategies in recent works on earning from arbitrary multimodal input data [67], we initially train the model on modality-complete samples and gradually expose it to more challenging examples with missing modalities. Specifically, the modality dropout
probability p(T) = min (Pmax: Tree
T-Tammap
Tesmmp *Pmax)
increases linearly with the epoch index after a
warm-up period Twannup and is capped at maximum value Pmax-
To introduce input-dependent dynamism based on modality combinations, we adopt the standard Sparse MoE [48] layer to process the input representation e. The MoE layer consists of lt experts-fully connected and router trainable R, which selects the top-k experts to process each of the L tokens. The progressive modality-dropout acts as form of regularization [60], encouraging implicit expert specialization without requiring any explicit auxiliary load-bat hod-bblancing losses. Such foss-free MoE optimization is also supported by recent works [23, 6. 56]. Our empirical observations (see Figure 4) suggest this specialization behavior: in a setting with ? experts and k = 1, the routing patterns of R exhibit variability across different combinations of input modalities at inference time. These results align with our design intent (as verified in Figure 6 in Section 4.2), indicating that MoEs can enable black-box specialization and dynamic routing based on input conditions. We use the final logit predictions from the selected experts (typically with & 1; for k > 1, we aggregate logits via averaging) to compute the task loss. Given aggregated logits ÿ E RC,, we first apply softmax normalization to obtain class probabilities
Comroeatio Compbaatio 2
Mocalla
1.0
0.5
0.6
0.4
0.2
Expert Activation IDs
Figure 4: Expert routing decisions across different input modality combi- nations, highlighting the implicit spe- cialization hehavior of the sparse MoF. layer.
5 exp(5.) For ground-truth label y E {L...., C}. the cross-entropy loss is: LCr: = log Py- Detailed implementation and optimization settings are described in Appendix C.
exp((-))
7

---

4 Results and Discussion
We evaluate MAESTRO against 10 state-of-the-art methods on four multimodal time-series datasets across three application domains, using per-segment accuracy and macro-F1 score.
Datasets. We leverage four multimodal time-series datasets spanning diverse sensing configurations and tasks. WESAD [46] includes 10 chest- and wrist-mounted modalities for 3-class cognitive stress classification. DaliaHAR, constructed from the Dalia dataset [45], is a 7-class physical activity recognition task using five physiological and motion modalities. DSADS [2] comprises 9-axis IMU signals from five body locations for 19-class activity classification. MIMIC-III, processed via MultiBench [25], is clinical dataset for 6-class diagnostic prediction using 17 modalities. Additional dataset statistics and preprocessing steps are detailed Appendix D.2.
Baselines. We compare MAESTRO against strong baselines spanning multivariate, multimodal, and missingness-awes aware approaches. From recent benchmark leaders [37], we include InceptionTimm [16], ResNetID [10], and Transformer [53], along with ITransformer [29]. which leverages inverted at- tention. For multimodal settings. we evaluate late fusion approaches including ensemble learning [5] and low-rank tensor fusion (LRTF) [30]. We further include MoF-based frameworks that support missing modalities-Flexxt
[67] and FuseMoE [15]-as as well as inoraction-centtiy models like MULT [52] and ShaSpee [55], motivated by [26, 25]. To evaluate robustness under modality dropout, we apply the missingness-awee aware scheme from Section 3.2.5 to Transformer, and compare against matively-missinnnss resilient baselines -aeelines-FlexMot. FuseMoE, and ShaSpec-undee varying amounts of missingness (10% to 40%). All baselines use published hyperparameters or undergo search when unspecified. Each dataset has three distinct splits (80% train, 10% valid and 10% test), with three trials per split. We report mean performance in the main text; full statistics and implementation details are given in Appendix E and D, respectively.
4.1 Advantages of MAESTRO: Superior Performance and Robustness to Arbitrary Missingness
Table 2 reports the performance of MAESTRO against multivariate and multimodal baselines under full modality availability, with the following key observations: (1) MAESTRO achieves consistent gains over the best baselines- relative improvement on WESAD, 5% on DaliaHAR, and 4% on DSADS; (2) the MIMIC dataset yields lower absolute performance and smaller gains due its task complexity-which aligns with previously reported findings [25]. To better interpret MIMIC results, we analyze macro-Fl scores; MAESTRO achieves 0.30, outperforming the generally strong baselines FuseMoE and FlexMoE (both at 0.27), reflecting an 11% relative improvement; (3) using symbolic representation of the multimodal input preserves semantic structure and improves the overall performance of the MAESTRO framework by 6% relatively under full modality observability; (4) overall, multimodal approaches outperform multivariate ones by an average relative improvement of 4% across all datasets. MAESTRO delivers consistent relative improvements-ae 8% over top multivariate and 4 over top multimodal baselines-acvve across all benchmarks.
Table 2: Performance (Accuracy/F1-scord) comparison across datasets under full modality observabil- ity. Best per dataset in bold and orange, second-best in italics and underlined. Mean_ reported.
Model Type
Model
WESAD
DaliaHAR
DSADS
MIMIC III
Acc f
F1
Acc 1
FL
Acc
F1
Ace t
F1
InceptionTime
0.59
0.51
0.73
0.68
0.81
0.81
0.29.0.01 0.74
Multivariate
Transformer
0.63+0.0L
0.5340.02
0.76
0.71
0.83
0.83
0.7840.02
0.22+0.00
ResNet1D
0.52.40.0
0.44
0.73ga00
0.69
0.79
0.78
0.77
0.18
Cross-variate
iTransformer
0.6740.c2
0.5840002
0.69
0.66gm.22
0.62.g..s
0.61 2.00
0.7740.01
0.14
Multimoda/MMM
LRTF
0.52
0.30
0.48
0.19
0.72
0.70
0.77
0.15
Ensemble
0.69 4.
0.57
0.74
0.71
0..9
0.57 20.1
0.28 0.28 10
0.27
FuseMoE
0.4740..8
0.41
0.7940.02
0.79-m.02
0.85
2.85
0.75
0.27
MM MoE.
FlexMoF
0.71
0.63
0.70
0.70
0.70
0.68
0.79:0.08
0.27:2.00
MM Interaction
MULT
0.60
0.42:xo02
0.72
0.72-m.a2
0.66ga.0s
0.65
0.79
0.21
ShaSpee
0.62
0.51
0.75
0.78
0.82
0.81
0.74
0.24
Ours
without SAX
0.69
0.82 0.55
on
0.84
0.78
0.77
0.30 0.79
with SAX
0.77
0.6644001
0.83
0.84
0.88
0.88
0.78
0.30
MAESTRO pragmatically designed to handle arbitrary modality missingness through reserved symbolic representations, adaptive attention budgeting, cross-modal learning, and modality dropout
8

---

0.73
0.90
0.93
0.36
0.63
0.75
0.73
0.31
0.52
0.60
0.53
0.26
0.45
0.33
(8)
(b)
[C)
(d)
0.32
0%
10%
30%
40%
0.30
10%
20%
40%
0.12
0%
10%
20% 30%
0.15
0%
10%
30% 40%
Missingness (%)
Missingness (%6
Missingness (%)
Missingness (%)
Teerafomerieeapplee)
FlexMoE
FuseMot
+
ShaSpec - Ours (with SAO
-a- Ours twithout SAX)
Figure 5: Comparative performance (Macro-FI score) of MAESTRO against missingness-awwe aware multimodal base-
lines and droppit--dooout--aeee levels (a) WESAD, (b) DaliaHAR, (c) DSADS, and (d) MIMIC-III.
Transformer shows its consistent superiority across varying missingness
training. As shown in Figure 5: (1) under 40% missingness, MAESTRO outperforms the strongest missingness-resllien multimodal baseline by an average relative F1 gain of 59% across datasets; (2) representative multivariate model-Trrnnffomne adapted with the same missingness-awe strategy as MAESTRO -enerallg outperforms other multimodal baselines, yet MAESTRO still exceeds its performance by 25% relatively; and (3) symbolic reservation for missing modalities in MAESTRO yields an average relative gain of 11 11% Across all datasets, MAESTRO achieves increasing accuracy improvements over the best baseline- -.66% absolute improvement under 10% missingness and 9.4% under 40 %. Detailed results are in Appendix E.
training
4.2 Architecture, Efficiency, and Empirical Case Study of MAESTRO
We further analyze MAESTRO through ablation and complexity studies on the WESAD dataset.
Ablation Study. We examine the key components of MAESTRO that drive its modality settings.
performance under both full and partial (40% missingness) As shown in Figure 6, symbolic tokenization, modality- specific positional embeddings, and sparse MoE routing each contribute substantial gains-ranghon from 5% to 22%. While modality dropout has limited impact under full modality (2%), it critical under missingness, where its removal causes a 9% drop. The 22% drop without the sparse MoE under incomplete observations further supports our design intuition that the router (R) implicitly specializes for different modality combinations, as illustrated in Figure 4 (Section 3.2.5).
Adayitt
Figure 6 Ablation of MAESTRO: absolute accurncy drop (A) from fall model, with ablated scores parentheses.
Complexity Study. To demonstrate the efficiency of MAESTRO, compute giga floating-point operations (GFLOPs), multiply-accumulate operations (MMACs) and number of trainable parameters under the complete- modality setting. From Table 3, we observe: (1) replacing sparse with full attention MAESTRO's 's components yields at most an absolute performance improvement of 3%, while reducing GFLOPs by an average of 20%: (2) mul- tivariate handling is slightly more efficient (0.4 GFLOPs lower) but suffers from nearly 10% lower performance; and (3) MAESTRO outperforms existing multimodal frame- works in both accuracy and efficiency for multimodal time- series. In particular, juxtaposing it against MULT under-
Table Computational Complexity. MAESTRO, sparse attention in the Per Modal and Cross Modal components is replaced by dense attention, referred as Full-Artn (Per-Modal) and Full-Attn (Cross-Modal), respectively. Replacing all spurse attention components with dense
attention denoted All Full Altn.
Midel
Ac.. MMAC| GELOP.
Parmmm (Mi
Malmaniar Modelt
iTrnchrmee
2413
573
13.87
"Tredfmmr
$.66
Malivedaa Moable
FooMiE
6534
1300
0.7
MULLI
28.55
MARSTRD
0:3708
2055
6:1
1.39
Am (he Modal
(180gg ss
1769
7.54
-325
8.41
All Aut--neniee (so Motto (178g18:
4393
8.78
1.39
scores the cost of pairwise-exhaustive modeling, which consumes nearly 200% more GFLOPs than our proposed MAESTRO.
Sensitivity Study. We present sensitivity studies for the model parameters- compereiss ratio (FF and expert count (?) Figure 7 (with details in Tables 16 and 17 in the Appendix), as well as for input noise. For input noise, we design two pilot experiments. In the first, shown in Figure 8, we show that the symbolic transformation not only enables input compression and symbol reservation (Section 3.2.1). 1), but also mitigates small local perturbations such as Gaussian noise. Next, a controlled study using simple additive noise (described in detail in Section E.3) in fixed modalities in Table 4, we observe that even for aggressive noise such as simulated electrical interference spikes, MAESTRO shows similar performance to full-modality scenario in one combination and in another its close to when the modalities are completely missing. This supports our multimodal treatment of time-series from heterogeneous sensors, with cross aatassaatentiin-baseeee design and Missingness-awee

---

Symbolic Tokenization that enables querying only the relevant inter- and intra-moda values for given task. This provides inherent robustness to missing modalities, and through these controlled experiments, we can also observe MAESTRO's s robustness to other simple perturbations. We also include preliminary time-dependent noise analysis (Section E.4 of the Appendix) to demonstrate MAESTRO's ability to handle asynchronous inputs, and high-density sampling study (Section E.5 of the Appendix) to highlight its computational benefits for long-range multimodal time series.
Table Performance under simple noise.
1.0
Comh. omits chest_acc and Comb. omits wrist BVP, EDA, and Temp.
8.9
Clean
Nolsy
Accursy
0.80
0.7
0.6
Corruption
Comb.
Comb.
0.4
None
0.77
0.77
0.70
Experts
0.70
Compression Ratio
0.3
input
Symbelic assion)
Symboic
Add, Noise (or 0.5)
Ramdom Noise
Missing
0.61
0.61
0.75
0.63
0.65
0.74
Add. Noise Spikes
0.00
0.73
Figure 7: Sensitivity Analysis.
Figure 8: Comparing the differen input representations with noisy inpul.
Several datasets exhibit large performance differences between the best and worst unimodal models 35% in WESAD, 59% in DaliaHAR). In WESAD (a 3-class task), the temperature modality performs near random, with an accuracy of 0.38. Full unimodal results are reported in Appendix F.1. Some datasets, like DSADS, exhibit higher redundancy, while in MIMIC, modality gaps are minimal, though overall accuracy remains low. These trends support our design intuition: modality heterogene- ity is inherent. MAESTRO addresses this diversity and remains competitive with the best unimodal models, offering average relative gains of 7%.
Case study on DaliaHAR. For activity recognition, motion sensors (wrist_ACe chest_ACO are clearly more relevant than other physiological sensors. We perform a combinatorially exhaustive study ( (A)) and observe that: (1) some modalities wwist_BVi and Frrist_Temp Leliss_Temp-feepee frequently category (accuracy S 0.70) (Table 5), and the averaged cross-
Temp-frequently
appear in the lower-performan modal attention map, c, from Section 3.2.4 uncovers such relational patterns (Figure 9); (2) a priori understanding of the most informative modalities further improves MAESTRO's performance from 0.83 to 0.85 average accuracy- accuracy-highlighting highlighting that an priori-guided MAESTRO can enhance predictive performance without any additional updates (complete results in Appendix F).
Table 5: Low (S0.70) va High (>0.77) performance combinations for DaliaHAR.
Low cow-performing Combinations
Acc.
fligh-performing Combinations
Acc.
0.530
obest ACC, EDA, wrist TEMP
chest_ACC, WrC
WESLACC, BVP, TEMP
0.67
ACC
0.79
0.00
ebestACC,, WVV
ACCC HVP, weist_ TEMP
0.67
chesLACC, WCC, EDA
shest,CCC, wris
0.79
0:50
vir.. ACC
15
wris wrC, wrist
WCC,
Vutiioenal Sequeece
ebestACC,, ACC, TERP
0.70
chesLACC
0.85
Figure 9: Averaged Atteation map for a batch across the maltimodal sequence.
Conclusion
We propose novel multimodal framework, MAESTRO, to model time-series from heterogeneous sens- ing modalities and address the limitations of existing multivariate approaches- such as oversimplified feature interactions and existing multimodal approaches, which often rely pairwise interactions or overemphasize a single dominant modality. MAESTRO achieves this by using symbolic represen- tations to tokenize both time-series data and missingness, applying adaptive budgeted intra-modal learning based on modality availability and relevance, and leveraging sparse cross- cross-modal attention to capture multimodal interactions. MAESTRO is specifically designed handle input dynamism, including arbitrary modality missingness. Extensive evaluations demonstrate MAESTRO's 's superior performance, robustness, and overall efficiency.
Broader Impact and Future Work. MAESTRO paves the way for more efficient and pragmatic handling of heterogenous sensing data. Currently, we handle complete missingness modalities. In in future, we aim to explore more advanced symbolic encoding strategies and extend the framework to address irregularly sampled and asynchronous sensing modalities.
6 Acknowledgment
We gratefully acknowledge support in part from the National Science Foundation under Grants 2038853, 2324936, and 2328973.
10

---

References
[1] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang. Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in neural information processing systems, 34:24206-2444 24221, 2021.
[2] Kerem Altun, Billur Barshan, and Orkun Tunçel. Comparative study on classifying human activities with miniature inertial and magnetic sensors. Pattern Recognition, 43(10):3605-3620, 2010.
[3] Anthony Bagnall, Jason Lines, Aaron Bostrom, James Large, and Eamonn Keogh. The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Mining and Knowledge Discovery, 31(3):606-66) 660, 2017. doi: 10.1007/ s $10618-016-0483-)
[4] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis- Philippe Morency. Multimodal machine learn- ing: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423 443, 2018.
[5] Murchana Baruah and Bonny Banerjee. Modality selection for classification on time-series data. Mile'SS 20:6th, 2020.
[6] Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint arXiv:2407.06204 2024.
[7] Sumit Chopra Kyunghyun Cho Divyam Madaan, Taro Makino. Jointly modeling inter- & intra-modality dependencies for multi-modal learning. In The Thirly-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: http:///wwweeeeeeeeee./o
id=XAKALZIGG.
[8] Bruno Dumas, Denis Lalanne, and Sharon Oviatt. Multimodal interfaces: A survey of principles, models and frameworks. In Human machine interaction: Research results of the mmi program. pages 3-26. Springer, 2009.
[9] Stéphane Dupont and Juergen Luettin. Audio-visual speech modeling for continuous speech recognition. IEEE transactions on multimedia, 2(3):141-151, 2000.
[10] Hassan Ismail Fawaz. 0rXiv:2010.0056,, 2020.
Deep learning for time series classification.
arXiv preprint
[11] Christos A Frantzidis, Charalampos Bratsas, Manousos A Klados, Evdokimos Konstantinidis, Chrysa D Lithari, Ana B Vivas, Christos L Papadelis, Eleni Kaldoudi, Costas Pappas, and Panagiotis D Bamidis. On the classification emotional biosignals evoked while viewing affective pictures: integrated data-mining- approach for healthcare applications. IEEE Transactions on Information Technology in Biomedicine, 14(2):309-318, 2010.
[12] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 15 15180-15190, 2023.
[13] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus. George B Moody, Chung- Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101(23):e215-2222 e220. 2000.
[14] Wenzhong Guo, Jianwen Wang, and Shiping Wang. Deep multimodal representation learning: A survey. leee Access, (16333-66339)) 2019.
[15] Xing Han, Huy Nguyen, Carl Harris, Nhat Ho, and Suchi Saria. Fusemoe: Mixture-of-experts transformers for fleximodal fusion. In A. Globerson, I. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 67850-67900. Curran Associates, Inc., 2024.
11

---

[16] Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petitjean. Inceptiontime: Finding alexnet for time series classification. Data Mining and Knowledge Discovery, 34(6):193-1192,, 2020.
[17] Wei- Bei-Bang/aa Jiang, Ziyi Li, Wei-Long Zheng, and Bao-Liang Lu. Functional emotion transformer for eeg-ase assisted cross-modal emotion recognition. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1841-1845. IEEE, 2024.
[18] Yinzhu Jin, Aman Shrivastava, and Tom Fletcher. Learning group actions on latent representa- tions. Advances in Neural Information Processing Systems, 37:127273-127295 2025.
[19] Wonse Jo, Ruiqi Wang, Baijian Yang. Daniel Foti, Mo Rastgaar, and Byung- Chenol Min. Cognitive based affective workload allocation for multihuman multirobot teams. IEEE Transactions Staman-Machin Systems, 2024.
[20] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[21] Elsa A Kirchner, Stephen H Fairclough, and Frank Kirchner. Embedded multimodal interfaces in robotics: applications, future trends, and societal implications. In The Handbook of Multimodal- Multisensor Interfaces: Language Processing, Software, Commercialization, and Emerging Penections-Vlluree 3, pages 523-576. 2019.
[22] Michelle A Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 International conference on robotics and automation (ICRA), pages 8943-8950. IEEE, 2019.
[23] Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. Uni-moe: Scaling unified multimodal Ilms with mixture of experts. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2025.
[24] Paul Pu Liang, Zhun Liu, Yao-Hung Hubert Tsai, Qibin Zhao, Ruslan Salakhutdinov, and Louis-Philiipe Morency. Learning representations from imperfect time series data via tensor rank regularization. arXiv preprint arXiv: 1, 2019.
[25] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. Advances in neural information processing systems, 2021(DB1):1, 2021.
[26] Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, et al. Quantifying & modeling multimodal interactions: An information decomposition framework. Advances in Neural Information Processing Systems, 01-27351-27399 2023.
[27] Paul Pu Liang, Zihao Deng, Martin Q Ma, James Y Zou, Louis-Philippe Morency, and Ruslan Salakhutdinov. Factorized contrastive learning: Going beyond multi-view redundancy. Advances in Neural Information Processing Systems, 36:32971-32999 2023.
[28] Jessica Lin, Eamonn Keogh, Li Wei, and Stefano Lonardi. Experiencing sax: novel symbolic representation of time series. Data Mining and knowledge discovery, 15:107-144, 2007.
(29] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625. 2023.
[30] Zhun Liu, Ying Shen, Varun Bharadhwajj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:186..00 000644 2018.
12

---

[31] Mengmeng Ma, Jian Ren, Long Zhao. Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2302-2310, 2021.
[32] Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust to missing modality? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18177-18186, 2022.
[33] Francisco Matos, Jorge Bernardino, João Durães, and João Cunha. A survey on sensor failures in autonomous vehicles: Challenges and solutions. Sensors, 24(16))5108, 2024.
[34] Manuel Meier and Christian Holz. Assessing the accuracy of cihoppeethysmographhe for wearable heart rate monitoring based body location and body motion in uncontrolled outdoor environments. Current Issues in Sport Science (CISS), 9())059-059, 2024.
[35] Manuel Meier and Christian Holz. Impact of optical wavelength the reliability of hootoolethysmographyy-basss heart rate measurements outside of controlled laboratory en- vironments. Current Issues Sport Science (CISS), 9(2):600-60, 2024.
[36] Manuel Meier, Berken Utku Demirel, and Christian Holz. Wildppg: A real-world ppg dataset of long continuous recordings. In A. Globerson, L.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 2246-2266. Curran Associates, Inc., 2024.
[37] Matthew Middlehurst, Patrick Schäfer, and Anthony Bagnall. Bake off redux: a review and experimental evaluation of recent time series classification algorithms. Data Mining and Knowledge Discovery, 38(4))9958-200) 2024.
(38] Payal Mohapatra, Akash Pandey, Sinan Keten, Wei Chen, and Qi Zhu. Person identification with wearable sensing using missing feature encoding and multi-stage modality fusion. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-2. IEEE, 2023.
[39] Payal Mohapatra, Vasudev Aravind, Marisa Bisram, Young-Joong Lee, Hyoyoung Jeong, Katherine Jinkins, Richard Gardner, Jill Streamer, Brent Bowers, Lora Cavuoto, et al. Wearable network for multilevel physical fatigue prediction in manufacturing workers. PNAS nexus, (10):pgae421, 2024.
[40] Payal Mohapatra, Shamika Likhite, Subrata Biswas, Bashima Islam, and Qi Zhu. Missingness- resilient Video-enhanced Multimodal Disfluency Detection. In Interspeech 2024, pages 5093- 5097, 2024. doi: 10.2 10.21457/mmesspechh2244 1458-
[41] Payal Mohapatra, Lixu Wang, and Qi Zhu. Phase-driven domain generalizable learning for nonstationary time series. arXiv preprint arXiv:2402.05966 2024.
[42] Payal Mohapatra, Akash Pandey, Xiaoyuan Zhang, and Qi Zhu. Can LLMs understand unvoiced speech? exploring EMG-to-text conversion with LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, 2025.
[43] Xuan Vinh Nguyen, Jeffrey Chan, Simone Romano, and James Bailey. Effective global approaches for mutual information based feature selection. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 512-521, 2014.
[44] Xiaomin Ouyang, Jason Wu, Tomoyoshi Kimura, Yihan Lin, Gunjan Verma, Tarek Abdelzaher, and Mani Srivastava. Mmbind: Unleashing the potential of distributed and heterogeneous data for multimodal learning in iot. In Proceedings of the 23rd ACM Conference on Embedded Networked Sensor Systems, pages 491-503, 2025.
[45] Attila Reiss, Ina Indlekofer, Philip Schmidt, and Kristof Van Laerhoven. Deep ppg: Large-scale heart rate estimation with convolutional neural networks. Sensors, 19(14):3079, 2019.
13

---

[46] Philip Schmidt, Attila Reiss, Robert Duerichen, Claus Marberger, and Kristof Van Laerhoven. Introducing wesad, a multimodal dataset for wearable stress and affect detection. In Proceedings of the 20th ACM international conference multimodal interaction, pages 400-408, 2018.
[47] Abhishek Sharma, Leana Golubchik, and Ramesh Govindan. On the prevalence of sensor faults in real- deployments. In 2007 4th Annual IEEE Communications Society Conference on Sensor, Mesh and Ad Hoc Communications and Networks, pages 213-222. IEEE, 2007.
[48] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gates mixture-of-experts layer. arXiv preprint wwXiv:1701.06530 2017.
[49] Satya Narayan Shukla and Benjamin M Marlin. Multi-time attention networks for irregularly sampled time series. arXiv preprint 21Xi::2001.10318. 2021.
[50] Haoqin Sun, Shiwan Zhao, Shaokai Li, Xiangyu Kong, Xuechen Wang. Jiaming Zhou, Aobo Kong, Yong Chen, Wenjia Zeng, and Yong Qin. Enhancing emotion recognition in incom- plete data: A novel cross-modal alignment, reconstruction, and refinement framework. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2025.
[51] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[52] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguistics. Meeting, volume 2019, page 6558, 2019.
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[54] Chenyu Wang, Sharut Gupta, Xinyi Zhang, Sana Tonekaboni, Stefanie Jegelka, Tommi Jaakkola, and Caroline Uhler. An information criterion for controlled disentanglement of multimodal data. arXiv preprint arXiv:2410.23990 2024.
[55] Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carneiro. Multi- modal learning with missing modality via shared- specific feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15878-15887, 2023.
[56] Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary- loss loss-free load balancing strategy for mixture-of-eexperts arXiv preprint 24X.v:2408.1566. 2024.
[57] Ruiqi Wang, Wonse Jo, Dezhong Zhao, Weizheng Wang, Arjun Gupte, Baijian Yang. Guohua Chen, and Syyng-Chholl Min. Husformer: A multimodal transformer for multimodal human state recognition. IEEE Transactions on Cognitive and Developmental Systems, 16(4):1374- 1390, 2024. doi: 10. 10.109/TCDS.22244335666
[58] Xue Wang, Tian Zhou, Qingsong Wen, Jinyang Gao, Bolin Ding, and Rong Jin. Card: Channel aligned robust blend transformer for time series forecasting. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https: / http///ppe/e//eeeeeeee net// forum?id=K.JssrOhurr.
[59] Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Yong Liu, Mingsheng Long, and Jianmin Wang. Deep time series models: A comprehensive survey and benchmark. 2024.
[60] Colin Wei, Sham Kakade, and Tengyu Ma. The implicit and explicit regularization effects of dropout. In International conference on machine learning, pages 10181-1018 PMLR, 2020.
14

---

[61] Chenwei Wu, Zitao Shuai, Zhengxu Tang, Luning Wang, and Liyue Shen. Dynamic modeling of patients, modalities and tasks via multi-modal multi-task mixture of experts. In The Thirteenth International Conference on Learning Representations.
[62] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2d-variation modeling for general time series analysis. International Conference on Learning Representations, 2023.
[63] Xingjian Wu, Xiangfei Qiu, Zhengyu Li, Yihang Wang, Jilin Hu, Chenjuan Guo, Hui Xiong, and Bin Yang. Catch: Channel-aware multivariate time series anomaly detection via frequency patching. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
[64] Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10): 2023.
[65] Zhen Xu, David R So, and Andrew Dai. Mufasa: Multimodal fusion architecture search for electronic health records. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 10532-10540, 2021.
[66] Haofei Yu, Zhengyang Qi, Lawrence Jang, Ruslan Salakhutdinov, Mouis-Philippe Morency, and Paul Pu Liang. Mmoe: Enhancing multimodal models with mixtures of multimodal interaction experts. arXiv preprint arXiv:23/ 1.0.11.0900 1.09580, 2023.
[67] Sukwon Yun, Inyoung Choi, Jie Peng, Yangfan Wu, Jingxuan Bao, Qiyiwen Zhang, Jiayi Xin, Qi Long, and Tianlong Chen. Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of-experts. In Advances in Neural Information Processing Systems, volume 37, pages 98782-98805. Curran Associates, Inc., 2024.
[68] Lei Zhang, Lei Zhang, and Du. Advances in multi-sensor data fusion: Algorithms and applications. Semsors, 10(2):771-7784, 2010. doi: 10.33900s100807771. URI. https: www. ncbi. mlm. nih. http://www....oeaaaaaaaaeeee/aaa
[69] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.
[70] Pengfei Zhu, Yang Sun, Bing Cao, and Qinghua Hu. Task-customized mixture of adapters for general image fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7099-7108, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims in the abstract and introduction are addressed in the paper
with thorough methodological and theoretical analysis in Section 3 and empirical evaluations in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed after the concluding remarks.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[Yes]
Justification: The main theoretical insights are discussed in Section 3.2.1 and a detailed derivation is given in Section A in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All the implementation details to truthfully reproduce the results are given in Section 3 and 4 of the main paper and Section D of Appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The codebase to generate the primary results is provided in the supplementary materials and upon the acceptance of the paper, will also be made publicly available. All the datasets used for this work are publicly available.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /niis.
gwwde// Caaaaae/dooSSSbbeeec oniolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the experiment details given in Section 4 and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All the statistical results with standard deviation as the error bar resulting from 3 independent runs of each experiment are provided the respective figures and additional details are given in Section and E of the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section D in the Appendix discusses all relevant computing requirements,
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //wuurius...../ cc Btt.lii//Ehhsssuuiiceioe
Answer: [Yes]
Justification:
Our research adheres to all the ethical guidelines outlined by NeurIPS.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We briefly discuss the broader impacts of our work in the concluding remarks and include expanded version in the Appendix G.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We propose a new technique to facilitate multimodal learning of time-series data from different sensors. Our contribution does not pose any explicit risks, pub- licly available datasets and employ end-to-end model training, which ensures transparency.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All dataset details and original authorship are cited in Section 4 and Section D.2 of the Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release any new datasets. The codebase for the MAESTRO implementation is provided as part of supplementary matcrials and will be released publicly with cleaner version under the creative commons (CC by 4.0) license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects, hence do not require IRB approval.
Guidelines:
21

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our work does not use LLMs for its core research contribution.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /tt////eurrr//e/eee/eee/e htt/e//rp..c//ooofeeeeeeeeeeeeee wwwpc//ce///nneeeeeeeeeoeoesseee http:pp/..c/o/oeeeeee/////////// for what should or should not be described.
22

---

Appendix
This Appendix includes additional details for the paper, "MAESTRO Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Series", including the reproducibility statement, additional details on symbolic tokenization and theoretical proof of Corollary 3.2 of the main paper (Section A), additional details for sparse multihead attention (Section B), additional experimental setup and training details of MAESTRO (Sections C and D) with detailed dataset introduction (Sec- tion D.2), more detailed results of the experiments shown in the main paper (Section E), additional experiments (Section F), and more discussion on broader impacts.
Reproducibility Statement
minimal source-code has been provided in the Supplementary Materials. We use public datasets and provide implementation details in the following sections.
A Additional Details on the Symbolic Tokenization
Piecewise Aggregate Approximation (PAA) and Symbolic Aggregate approXimation (SAX) are sequential time-series compression methods that reduce temporal resolution by dividing the signal into fixed-size windows. PAA summarizes each window by its mean value, producing a smoothed, real-valued lower-dimensional representation. SAX builds on PAA by further discretizing these means into symbolic tokens using breakpoints derived from standard Gaussian distribution, resulting in compact and interpretable symbolic sequence. As detailed in Section 3.2.1 of the main paper, PAA serves as an intermediate step in the SAX transformation pipeline, bridging raw signal compression and symbolic quantization. While PAA captures coarse temporal structure, SAX enables symbolic reasoning, efficient indexing, and explicit missingness encoding via reserved symbol. An illustration for their comparison is shown in Figure 10.
0.5
0.0
Magnitude
-0.5
-1.5
Original Time Series
-2.0
PAA Reconstruction
-2.
Reconstruction
Time
Figure 10: Comparison of PAA and SAX representations. The raw photoplethysmograph (PPG) signal æ[t] is first segmented and averaged into [[w] via PAA. SAX extends this by mapping each PAA segment discrete symbol s s[w] E {s1,.... 8a). yielding a compact symbolic sequence. reserved token can be used to indicate missing segments.
Proof Sketch for Corollary in Section 3.2.1 (Cross-Modal Relational Preservation)
Goal. Show that:
Dise (s), +()
Dist Syst (s"",88)
s <
| x2ll22 1,
xx 12
Ej + cm-
Assumptions.
1. Lower-bound property of SAX (MINDIST):
Dist (5/, s < | =fll2,
Distin (8. 5 ||0 xx 11.
2. Bounded symbolic approximation error:
|
2 2| Dist (5,, S Eg,
|| x 21ll2 2
Dist , (x cm
23

---

Proof. We begin by applying the triangle inequality, which states that for any real numbers a, b, the following holds:
aa - S + cl.
We use this property iteratively to decompose the difference between the symbolic distances.
Dist km
Dist.m sym
Distlym - ||
xz ll2 + || 2/ll2 || 2x ll2 + -|=0""
2x 112 Dist
Dist sym
|| 112| + ||o? - =xll2 - 1|,07"" - xx 112 + |||=?" xx 222112
Dist.
-
By the assumption of bounded symbolic error, we have:
Dist sym - || o
11
S ej.
Distyy - ||=,"" xx 1|2 S cm
Substituting these bounds, obtain:
Dist Dist S || - 2x112 2 |> - 22112 + Eg + Cm
B Additional Details for Sparse Multihead Attention
We adopt the sparsity measurement proposed by Zhou et al. [69] to efficiently identify dominant queries without computing all query-key pairs. For each query vector q Q, compute its sparsity score P(q, K") over a sampled subset of keys K" c K where 'Ko = u log Lx:
qk Vd
Σ qk
P(q, K')
max KEK'
KEK' Vd
This max-mean measurement evaluates the query's attention diversity by comparing its maximum alignment with the average alignment over the key subset. Queries with higher P(g. K") scores contain more distinctive information and are prioritized in our sparse attention mechanism.
The top-u queries with v u ulog LQ highest scores are selected for full attention computation, reducing the complexity from (LLQLk) to O(u log LQ log Lx). This adaptive selection enables efficient processing while preserving the most informative query-key interactions.
C Training and Optimization Details
Ubuntu OS server equipped with NVIDIA TITAN RTX GPU cards using Py PoTch framework. Every experiment is carried out with different seeds (2711, 2712, 2713). During model training, use Adam optimizer with learning rate from le-5 to 1e-3 and maximum number of epochs set to 150 based on the suitability each setting. We tune these optimization-rellted hyperparameters for each setting and save the best model checkpoint based early exit based on the minimum value of the loss function achieved on the validation set.
All experiments are performed
Modality Dropout Scheme based on Curriculum Learning. As described in Section 3.2.5, the modality dropout illustrated Figure 11.
Hyperparameters. The key hyperparameters in MAESTRO are listed in Table 6 and are kept fixed across all datasets in this paper. However, more comprehensive hyperparameter search could potentially yield further improvements in performance.
D Experimental Setup Details
D.1 Performance Metries
Accuracy. Given N samples, Accuracy defined as the proportion of correct predictions:
Accuracy
Kr = =v)
24

---

0.5
Modality Dropout Rate)
0.4
0.3
0.2
0.1
0.0
10
20
30
40
50
Epochs
Figure 11: Modality Dropout Scheme based on Curriculum Learning (supporting illustration for
Section 3.2.5).
Table 6: Model design components and hyperparameters.
Design Component
Hyperparameter
Value
Symbolic Tokenization
Number of alphabets
20
W Compression factor
Max Attention Budget
B
Sparse MoB
Ω Number of experts
k Selected experts per token
Macro-F1. Re:
Let be the set of classes. For each class c C, we compute precision P. and recall
R.. Macro-F1
F1 P. + R.
ΣFIe Flc
|C|
Relative Improvement. We report relative measurements for all metrics follows. Let Metriebase be the baseline performance and Metricour be our model's 's performance. The relative improvement is:
MetriCours Metrictase
Relative Improvement
100%
MetriCkase
We report absolute improvement in the case of accuracy-based based metrics as the direct difference between our model's performance and the baseline performance, i.c.:
bbsolute Improvement.
Absolute Improvement Metric Metrichase
D.2 Dataset Details
In this section, we present detailed information about the datasets used for evaluation, including class distributions, overall statistics, and preprocessing steps.
WESAD. We directly leverage the synchronized data from [46] for the WESAD dataset. The modalities and their corresponding sampling rates are summarized in Table 7, and the class distribution shown in Figure 12.
DaliaHAR. We adapt the DaLiA dataset for activity recognition using multimodal sensor data, addressing the scarcity of datasets that offer both diverse modalities and fine- ine-grained activity labels. Figure 13 illustrates the raw distribution activity labels from a single subject recording. To preprocess the data, first segment the continuous recordings based provided in the annotation files. Non-informative segments such as baseline and no-activity periods excluded. We then apply sliding window approach with window size of 8 seconds and 2-second overlap. Since all sensor streams are temporally aligned, each modality is segmented consistently, and each window is assigned the corresponding activity label for that subject.
absolute timestamps
The resulting processed data summary is given Table 8 and the class distribution is shown in Figure 14.
25

---

Table 7: WESAD dataset modality details.
Modality
Sampling Rate (Hz)
Variates
7000
chest_ACC
700
3
6000
chest_ ECG
Che
700
chest_EMG
700
4000
chest_ RESP
700
3000
@hest_EEA4
700
I
2000
ThMP
700
wrist_AC
32
3
wrist BVP
64
Class Labels
wrist EDA
4
Figure 12: Distribution of the classes for WESAD dataset.
wriMt_TEMP TEMP
Output Classes: Baseline, Stress, Amusement
Acceleratton
LINCH
Figure 13: Visualizing the raw accelerometer data from wrist for the Dalia dataset.
12000
10000
Modality
Sampling Rate (Hx)
Variates
Total Count
8000
chest_ACC
700
wrist_ACC
32
wrist BVP
64
6000
EDA
wrist TEMP
4000
Output Classes: STAIRS, SOCCER CYCLING, DRIVING, LUNCH, WALKING, WORKING
2000
Table 8: DaliaHAR dataset modality details.
3
4
5
Class Labels
Figure 14: Distribution of activity classes in the DaliaHAR dataset.
DSADS. For the DSADS dataset, follow the original preprocessing steps and report the cor- responding sensor modalities along with their specifications in Table 9. The class distribution is presented in Figure 15.
MIMIC. We adopt the preprocessing for the MIMIC dataset as defined by the multimodal bench- marking suite, MultiBench [25], to comply with standardized benchmarking practices. Overall, our performance-showe in Table Section 4.1 of the main paper-aligny with the results reported in the benchmarking suite. However, we observe clear class imbalance, as illustrated in Figure 16. Therefore, we also report the Macro-F1 score, which is particularly more informative than accuracy for the MIMIC datasct.
26

---

Multimodal Baselines. For the LRTF and MULT baselines, we implement them following the their original implementations and Sampling Rate (H2)
Modality
Variates
torso
25
17.5
right_an
25
25.0
left_arm
25
12.
right, leg.
25
3000
left log
25
Ourput Classes: Sitting. Snanding, laing on back, Tyrnng on right side, Ascending statrs, Descending stairs, Standing in an elevator (still), Moving around in an elevator, Welking in parking lot, Walking treadmill (faat) Wolking on (inclined), Running treadmill, Exercising
stepper. Exercising on cross trainer, Cycling on exercise bike (horizontal), Cyeling exercise bike (verticul), Rowing, Jumping, Playing basketball
Class
Figure 13: Class distribution of DSADS ac- tivity labels.
Table 9; DSADS dataset modality details.
Modality
Sampling Rate (HFx)
Varlates
glas gow IP HR Temp oay
urinc whe
2
20000
3300
bdc2
Na
5000
Bil
Age
Clors
ied9
hem, mai
Figure 16: Distribution of ICD-9 class labels the MIMIC-III dataset.
adm_type
Output Classes: diagnostic
calegories (cogrse-raoine))
Table 10: MIMIC-III dataset modality
details.
D.3 Baseline Implementation Details
Multivariate Baselines. We follow the MTS- [37] implementation for Inception'ime and ResNetID. For the Transformer, we adopt the standard implementation with 8 heads and layers, including positional encoding. In the case of the missingness analysis in Figure 5 of the main paper, we adapt the Transformer by retraining it using the same modality dropout scheme as employed in MAESTRO, enable a fair comparison with multivariate baseline that is natively robust to missing data. This modification allows the adapted Transformer to serve as a competitive baseline, as evidenced by the performance trends shown in Figure
MultiBench [25] framework. For the remaining baselines, open-source resources for reproduction.
In the robustness study in Section 4.1 in the main paper, we include only those baselines that natively support missingness, along with the adapted Transformer. We train these models using their original settings and evaluate them samples with dynamically missing modalities. Specifically, we randomly drop out modalities with increasing severity, ranging from 10% to 40% and report the performance.
E
More Detailed Results of the Main Paper Experiments
This section includes more detailed results from those experiments discussed Section of the main paper.
27

---

E.1 Robustness Results
The primary results using the Macro-F1 performance metric for varying levels of missingness across all datasets are presented in Figure 5 in Section 4.1 of the main paper. Tables 11, 12. 9. and 14 provide the complete statisties for accuracy and Fl scores for all Brasettt-WSAAD. DaliaHAR, DSADS, and MIMIC-III, respectively.
Table 11: Accuracy and Fl-score (meanga) across different missingness levels for the WESAD dataset (supporting results for Figure 5 in Section 4.1 in the main paper). Model
0%
10%
20%
30%
40%
Acc
Ace
Acc
Acc
F1
Transfomeer
0.670. 04
0.53g.os
0.63g 09
0.456 on
0.62p.00
0.49% os 0.61p.o4
0.48g.o0 oa 0.61o as
0.46p..1
FlexMol:
0.71o.00
0.640.11
0.68g os
0.61o.08
0.65g.06
0.58%
0.62o.os
0.56g os 0.596.or 07
0.52o.07
FuseMoF
0.48o.18
0.400.00
0.46g.02
0.410.03
0.45p.
0.40,.
0.410.00
0.38g.04
0.390.
0.370.02
ShaSpec
0.65o.5e
0.55g 0.54g.44
0. 0.496.44
0.53p.50
0.47g.44
0.48p.46
0.44g.41
0.436.12
0.41o.38
Ours(wlo SAX)
0.690.13
0.55g.00
0.68o.11
11
0.546.
0.670.09
0.53g 04
0.650.os
0.52p.04
0.666.08
0.530.06
Ours
0.77o.07
0.666..03
0.77g.06
0.680.04
0.54g.04 0.74p.07
0.740.07 0.64g.06
0.7 0.610.06
Table 12: Accuracy and Fl-score (meanga) across different missingness levels for the DaliaHAR dataset (supporting results for Figure in Section 4.1 in the main paper).
Model
0%
10%
20%
30%
40%
Acc
F1
Ace
F1
Ace
F1
Acc
F1
Acc
F1
Transformer
0.72o.
0.70o.08
0.69p.04 04
0.670.07
0.67p.04
0.65g.07 0.640.06
0.62g.0s
0.61o.n6
0.59o.os
FlexMoE
0.70u.
0. 0.700.0s
0.63g.04
0.6lo.as
0.53p.04
0.51o.06
0.45o.01
0.43p.02
0.37o.05
0.35p.o3
FuseMoo
0.78o
0.796.03
0.67
0.680.03
0.58g.01
0.580
0.480.03
0.46g
0.40o.04
0.360.03
ShaSpec
0.740.
0.770.02
0.64g.02
0.670.03
0.55
0.57o.02
0.480.02
0.47g.01
0.440.02
0.390.01
Ours(w/o SAX)
0.820.n8
0.84p.05
0.80g.os
0 0.830.02
0.770.04
0.790.0s
0.736.05
0.76g.02
0.71o.02
0.730.01
Ours
0.83o.01 01
0.84p.01
0.83p 64 0.856.03
0.81g.0a
0.82g oa. 0.786.03 0.79g.oa
0.746.03 03
0.75p.0a
Table 13: Accuracy and F1-score (meansa) across different missingness levels for the DSADS dataset (supporting results for Figure 5 in Section 4.1 in the main paper).
Model
0%
10%
20%
30%
50%
Ace
Fl
Acc
F1
Acc
Fl
Acc
F1
Acc
Fl
Transfommer
0.88o. p4
0.88p.01
0.78a.13
U.77
0.74p.
0.73a.14 0.71o.33
0. 0.66g 0.70g.15
1o
0.68p.os
FlexMoB
0.67o.01
0.03e.o3
0.46g.05
0.440.o5
0.35p.09
0.34g.
0.25e..0
0.23p.01
0. 0.190.
0.177.o7
FuscMoE
0.840.
0.85p.05
0.59
0.6lo.az
0.46p.04
0.460.04
0.430.02
0.44g.0s
0.37a.
0,38p.o3
ShaSpec
0.8
0.796.00
0.58g
0.580.03
0.45p.o2
0.450.02
0.296.01
0.27g.02
0.22o.01
0.20o.o2
Ours(wlo SAX)
0.78o.02
0.770.02
0.790.00
0.77o.01
0.76
0.75g.
0.720.01
0.71g.01
0.660.a6
0.650.04
Ours
0.89u.01
0.88p.01
0.86g.oz
0. 0.86p.01
0.84p.00 0.84g.01 0.83p.01 0.82p.01
0. 0.790.
0. 0.790.02
Table 14: Accuracy and F1-score (mean across different missingness levels in MIMIC dataset (Supporting results for Figure 5 in Section 4.1 in the main paper).
Model
0%
10%
20%
30%
40%
Acc
F1
Acc
F1
Acc
Fl
Acc
F1
Acc
F1
Transformer
0.780.02
0.78g.o4 0.22p.01
0.21o.03
0.78p.o1
0.22g.02
0.77o.01
0.22g.o1
0.77a.04 0.21o.on
FlexMoE
0.79g.02
0,276.03
0.78g oz
0.25
0.25g.01 0.79p.o1
0.78p.m
0.24g.02
0.78g.01
0.21o.01
FuseMoE
0.76o.
0.26o.00
0.750.on 01
0.27o.o1
0.73p.o2
0.236 no
0.726.03
0.23p.01
0.72o.02
0.22o.o1
ShaSpec
0.76u.
0,25p.01
0.
0.250..z
0.74p.01
0.23o.
0.740.00
0.20p.02
0. 0.70u.
0.21o.on
Ours(w/o SAX)
0.80g.
0.31o.o1
0.80g
0.31o.a2
0.77p.o1
0.28o.
0.796.02
0.27g.01
0.77o.02
0.26p.o2
Ours
0.796.
0.30p.oz
0.78g
0.300
0.76p
0.290
0.78p..1
0.27g.01
0.77o. 0.25p
E.2
Ablation Results
The complete statistics of the Figure 6 in Section 4.2 of the main paper is given in Table 15,
T Additional results on the sensitivity to hyperparameters w and Z experts in the Sparse-MoE layer, across two compute platforms RTX A6000 and Jetson TX2.
E.3 Simple Additive Noise
In the current work, we are focused on handling complete modality missings, which is a common issue in real-world sensing applications with large number of modalities (N > 4), where sensor
28

---

Table 15: Accuracy (meanga) on the WESAD dataset under full data and 40% missingness conditions for an ablation analysis (supporting results for Figure 6 in Section 4.2 of the main paper).
Method
Full
40 Missing
Ours
0.78u.07
0.71
w/o Symbolic Transformation
0.690.13
0.66g.07
w/o Modality Embedding
0.70g.
0.55
w/o Modality Dropout
0.76g.09
0.62
w/o Adaptive Attn Budget
0.7
0.67
w/o Cross-modal SparseMoF
0.70g.os
0.50p.01
Table 16: Performance comparison across number of experts.
Experts
ACC
RTX A6000
Jetson TX2
Latency (ms)
GFLOPs
Latency (ms)
GFLOPs
8
0.80
352.09
7.15
1298.52
6.16
0.79
279.41
7.08
1315.08
6.12
0.75
245.03
7.07
1291.25
6.00
fails and we are left with an arbitrary set of multimodal inputs make predictions. We also include preliminary study on noise robustness, which focuses on robustness to missing modalities, it could be a promising direction for future work on MAESTRO. We have conducted some initial exploration with three types of noise defined as follows. Here & denotes the subset of modalities chosen for introducing corruption:
1. Random noise: Selected channels are replaced with i.idd Gaussian noise, x 6, (0, o²), for e C
2. Additive noise: Gaussian noise added the original signal, x = x 6. -x0,,e~(0,,0+)) for e C
3. Additive noise with spikes (representing electrical spikes): Sparse sigg-magnttude impulses (spikes) are added in addition to Gaussian noise:
X = = e M <~~(0,00))
where M Bernoullip)) is a binary mask indicating spike locations with p probability, S E { (mm +m) is the spike magnitude, @ denotes element-wise multiplication.
We present the zero-shot performance MAESTRO on noisy WESAD data in the following two combinations: in combination 1, we dropped the chest accelerometer, and in combination 2. we dropped the wrist BVP, EDA, and Temp. Please refer Table 6 in our Appendix for more details on the WESAD modalities.
Also, in Figure 8, the Symbolic representation without compression refers only using word length of and only quantizing the time-series values to a fixed vocabulary.
E.4 Pilot experiments on Asynchronous Modalities
We present a brief pilot study in a simplified setting to emulate asymnchronous input data partial- missingness of modality. We mask approximately 25% the samples from the beginning and end of the sequence in 20% of the modalities (in this case, the wrist BVP, TEMP, and EDA) during inference to demonstrate zero-shot transfer capability. Currently, the model treats this partial missingness similarly or slightly better than complete missingness.
This performance could potentially be improved by incorporating standard techniques, such as absolute positional encoding, or by leveraging prior models like mTAN [49], which can more effectively encode irregular time-series with semantic meaning to support downstream cross-modal learning. Additionally, the generalization of the MAESTRO framework to distribution shifts- shifts-possibly arising from upgrades to one of the sensing modalities-com be enhanced using low-overhead design strategies such PhASER [41].
29

---

Table 17: Performance comparison across varying word lengths for SAX.
Word Length
ACC
RTX A6000
Jetson TX2
Latency (ms)
GFL.Ps
Latency (ms)
GFLOPs
0.80
279.11
7.08
1315.10 10
6.12
4
0.77
270.00
3.56
677.81
3.06
0.80
275.69
1.74
349.33
1.50
Table 18: Zero-shot transfer accuracy for MAESTRO under partial modality missingness scenarios.
Modalities
Accuracy
All available
0.77
Completely missing 20% modalities
0.61
Mask first 25% time steps in 20% modalities
0.60
Mask last 25% time steps in 20% modalities
0.65
E.5
High density input samples
We resample the modalities in WESAD to increase the sequence length in order emulate high- frequency sensor readings and report the performance below. As expected, our proposed MAESTRO performs consistently even with longer sequence lengths. However, upon replacing the sparse attention with canonical dense self-attention layers in both the intra-modal and cross- modall stages, encounter mem-ooy (OOM) issues at higher sampling rates (in this case, 128 Hz), highlighting MAESTRO's s advantage resource efficiency. All experiments were conducted on the RTX A6000.
Table 19: Comparison of MAESTRO and dense attention models under different sampling rates. OOM out-of-memory. Sampling Rate
MAESTRO GFLOPs
Dense Attention
Acc
MMACs
Acc
GFLOPs
MMACs
32
0.77
6.13
3066
0.75
8.78
4205
64
0.77
14.41
7205
0.71
23.02
11510
128
0.73
29.01
14502
OOM
OOM
0OM
F Additional Experiments
In additional the experiments shown in the main paper and Section also conducted additional experiments to further evaluate our approach, as shown below.
Unimodal Sweep Results
This section presents the unimodal results for WESAD, DSADS, DALIAHAR, and MIMIC, as shown in Table 20. For unimodal training, we train individual models for each modality using the Transformer backbone. These results indicate that in some applications, the modalities contain redundant information as seen in DSADS-Whnt -whree unimodal performance is not significantly lower than multimodal performance. In contrast, in datasets like Dalia, certain modalities perform close to random guessing (c.g., wrist_TEMP with 0.38 accuracy for 3-class classification task). Since MIMIC shows overall lower performance, we report the top five modalities and additionally provide the F1 -corre to highlight the performance boost achieved by using MAESTRO (with an Fi-score of 0.30), compared unimodal models which typically achieve around 0.15 F1 in most cases.
F.2 Supporting Results for Case Study on DaliaHAR from Section 4.2
Based on known priori, we evaluate three scenarios: 1) using the two best modalities-wwrstt and chest_ ACC-we we train bimodal model, 2) we run inference on the adapted Transformer using only these two modalities while dropping the rest, and 3) we evaluate MAESTRO the presence of only these two modalities.
30

---

Table 20: Unimodal performance across all datasets (supporting results for Section 4.2).
Dataset
Modality
ACC
STDEV
Torso
0.63
0.01
Right Arm
0.74
0.02
DSADS
Left Arm
0.85
0.03
Right Leg
0.81
0,02
Left Leg
0.83
0.01
wrist ACC
0.81
0.04
wrist_BVi BVP
0.50
0.04
Dalia
wrist_EDA
0.45
0.03
wrist_TEMP
0.35
0.03
chest ACC
0.85
0.01
chest_ ACC
0.67
0.04
chest_ECC
0.75
0.13
chest_EMC
0.75
0.01
chest_RESS 0.63
0.04
chest_ EDA
0.60
0.04
WESAD
chest TEMP
0.71
0.02
wrist ACC
0.66
0.12
wrist BVP
0.71
0.08
wrist EDA
0.60
0.04
wrist TEMP
0.38
0.12
glasgow
0.77
0.01
BP
0.72
0.08
MIMIC
HR
0.72
0.07
Temp
0.77
0.01
oxy
0.76
0.01
Table 21: Fl score and standard deviation for 5 best MIMIC modalities.
Modality
FT Score
STD
glasgow
0.17
0.02
BP
0.17
0.02
HR
0.16
0.02
Temp
0.15
0.01
oxy
0.16
0.01
Our results, shown in Table 22, highlight that MAESTRO with a priori knowledge at run-time performs competitively (with absolute improvement of 3%) compared to end-to-end model trained on these pre-selected modalities. This demonstrates that MAESTRO is capable of effectively modeling multimodal interactions and learning task-dependent semanties from multimodal data.
Table 22: Accuracy with priori modality mask inference: TEMP, chest_ACC] [1, o, o, O, 1] (supporting results Section 4.2).
wVist_BVP, wrist_EDA,
Method
Accuracy
Bimodal (wrist chest)
0.82
Transformer (adapted)
0.71 10.01
MAESTRO
0.85
E33 Visualization of Unimodal Representations for DaliaHAR
To further support the results in Table 5, we plot the t-SNE projections of the unimodal models' latent representations for each modality in DaliaHAR, shown in Figure 17.
31

---

WISLACC
WESE BVP
Wist BDA
WIst TEMP
cCC
Figure 17: t-SNE projections of the unimodal models' latent representations for each modality in DaliaHAR (supporting results for Table 5 in Section 4.2 the main paper).
G Broader Impacts
MAESTRO paves the way for more efficient and practical handling of heterogeneous sensing data. has the potential to enhance analytics and, in turn, the performance of ubiquitous sensing applications across diverse domains-including smart home monitoring, daily living assistance, fitness and wellness interventions, elderly care, healthcare, and environmental monitoring. These applications rely on rich, continuous streams sensory data, where sensor reliability often cannot be guaranteed. For example, in environmental monitoring within remote or inaccessible locations, sensors may fail due to power loss or harsh conditions. In such scenarios, maintaining robust performance with only a subset of available modalities is critical.
Currently, MAESTRO addresses complete modality-level missingness, demonstrating its ability to sustain model effectiveness even under challenging sensing conditions. In future, we aim to explore more advanced symbolic encoding strategies and extend the framework to address irregularly sampled and asynchronous sensing modalities and
32


Reviews:
[{"review_id": "ESka1MqMpk", "reviewer": "Reviewer_ALza", "summary": "The paper considers the following problem: given M modalities where each modality is a multi-dimensional time series, the objective is to learn a label function that maps from the M modality time-series space to the label space of size C. The main challenge considered is that some of the modalities may be missing, and the labeling function is expected to be robust over arbitrary missing modalities. To this end the paper proposes the following approach:\n1. Aggregation: applying piecewise aggregation with a window size T/W compressing the sequence length from size T to W.\n2. Symbolic tokenization: mapping the compressed representations  into a symbol from a range of alpha values. One symbol is reserved to indicate missingness. The mapping is performed in a way that preserves the approximate distances before and after tokenization (Cor 3.2).\n3. Feature extraction: using an encoder for each modality, based on a sparse self attention mechanism.\n4. Cross modal learning: achieved by concatenating features along the temporal dimension to form a unified sequence. To overcome the L^2 attention complexity (especially after concatenation with large sequence length), the paper uses sparse cross attention achieving a complexity of L log L.", "questions": "Could the authors please provide intuition on how the proposed algorithm overcomes the exponential growth of missingness patterns?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow.\n- The proposed method is robust to any missing modality pattern which is challenging as the number of missing modalities combinations grows exponentially with the number of modalities.\n- The authors algorithm introduces multiple operations to reduce computational complexity including: aggregation to decrease length from T to W, sparse self attention, and sparse cross attention.\n- The numerical results show big improvements over state of the art for multiple datasets.\n\nOverall, my impression  on the paper is positive, however, due to the following concerns I recommend a borderline accept.\nWeaknesses:\n- My main concern is regarding the limited novelty of the paper. Even though the results are good, the algorithm and techniques only combine those of existing works.\n- The algorithm seems adhoc without sufficient explanation on why such approach could overcome the exponentially many missingness patterns challenge.\n- Minor: the algorithm contains several hyper-parameters (the size W, alpha, ...) without any theoretical guidance on how they can be tuned, which introduces a lot of overhead in training.", "was_revised": true, "final_justification": "My impression remains positive about the paper, however, I keep my score at 4 since I recommend acceptance but would also be ok otherwise due to the lack of novelty as the algorithm mainly combines existing approaches.", "rebuttal": "Thank you for your encouraging feedback on our work's pragmatism, performance and robustness. We provide our response below :\n\n> Even though the results are good, the algorithm and techniques \nlargely combine those of existing works. \n\nWe acknowledge that prior advancements in the fields of data mining [1] and long-horizon time-series forecasting [2] have inspired us; but our contribution lies in adeptly leveraging these ideas to solve a different problem--learning from dynamic and heterogeneous multimodal time series. We propose adapting the sparse attention technique at inter- and intra-modal stages with a learnable attention budget (the gating function in Section 3.2.3); using the opportunity with discrete symbols in tokenization to reserve a symbol for missingness (Section 3.2.1); and incorporating Mixture-of-Experts (MoE) to allow for implicit expert specialization under arbitrary modality combinations (Section 3.2.5)—which significantly extend beyond direct combinations of existing techniques. Our ablation study in Figure 6 shows the effectiveness of these proposed techniques.\n\n\nAdditionally, our work presents an alternate viewpoint of multimodal handling of multi-sensor data over the commonly adopted multivariate modeling, which is often chosen because the sensing modalities share the same (numerical) representation in the input space. Our design considers the heterogeneity in the sensors and proposes an effective multimodal learning framework, MAESTRO.\n\n---\n\n> Justification of how the algorithm overcomes the many missingness patterns challenge.\n\nWe conduct an ablation of all our proposed components under complete observation and under modality missingness to demonstrate their roles (in Figure 6 of Section 4.2). The key intuition is that cross-modal attention (which improves performance by 21%) allows each modality to adequately query the necessary modalities—including itself—based on their availability and relevance. The modality dropout augmentation (improves performance by 9%) and the modality position embedding (improves performance by 16%), which denote the ordering of the modalities, help facilitate this learning. Supporting cross-modal learning with modality-specific embedding and augmentation helps MAESTRO achieve consistently superior performance (an average improvement of 9%) over other baselines, even under 40% missingness.\n\n---\n\n> Minor: The algorithm involves several hyperparameters (e.g., window size $W$, $\\alpha$) without theoretical guidance for tuning, adding training overhead.\n\nThank you for commenting on this aspect. In our current work we have not extensively fine-tuned our hyper-parameters (as shown in Table 5 of the Appendix), yet we are able to outperform the existing baselines under complete and arbitrary modality combinations. However, for application-specific scenarios, based on the available computational resources, these hyperparameters can be selected to optimize for performance and computational efficiency. We present a brief sensitivity analyses for number of experts and the word length for the WESAD dataset across two computational platforms below :\n\n##### Varying Experts for MoE (default w = 2)\n\n\n| # Experts | ACC | RTX A6000 - Latency (ms) | RTX A6000 - GFLOPs  | Jetson TX2 - Latency (ms)   | Jetson TX2 - GFLOPs |\n|-----------|----------|--------------------------|---------------------|-----------------------------|---------------------|\n| 8\t\t    |   0.80   |     352.09\t              |       7.15          |\t 1298.52\t              |      6.16           |\n| 4\t\t    |   0.79   |     279.41\t              |       7.08          |\t 1315.08\t              |      6.12           |\n| 2\t\t    |   0.75   |     245.03\t              |       7.07          |\t 1291.25\t              |      6.00             |\n\n\n##### Varying Word Length for SAX (default w = 2)\n\n| Word Length (w) | ACC | RTX A6000 - Latency (ms) | RTX A6000 - GFLOPs | Jetson TX2 - Latency (ms) | Jetson TX2 - GFLOPs |\n|------------------|----------|---------------------------|----------------------|-----------------------------|-----------------------|\n|2\t\t           |  0.80    |  279.41                   |\t  7.08\t             |        1315.08\t            |       6.12           |\n|4\t\t           |  0.77    |  270.00                   |\t  3.56\t             |        677.81\t             |      3.06           |\n|8\t\t           |  0.80    |  275.69                   |\t  1.74\t             |        349.33\t             |      1.50          |\n\n\nWe will include this analyses in our Appendix.\n\n---\n\n[1] Experiencing SAX: a Novel Symbolic Representation of Time Series, 2007\n[2] Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, 2021"}, {"review_id": "hwRA0CpVxM", "reviewer": "Reviewer_hWHy", "summary": "The authors propose a new model (tokenization + encoding + prediction) scheme for multi-variate multi-modal time series that handles missingness and variable length sequences. The method tokenizes everything into discrete token sequences and then uses sparse attention mechanisms to deal with the long sequence length that results from concatenating the representations of multiple modalities. Experimentally, the authors compare 11 methods on 4 public datasets that physiological/health measurements.", "questions": "section 3, maestro.\n\n- In the signal tokenization:\n\n  - first step is clear: average each signal in windows\n\n  - next step is unclear. \"Similar to Lin et al. [28], we use equidistant Gaussian breakpoints to partition the range into α equiprobable regions, mapping each xˆ[w] to a symbolic token s[w] ∈ {s1, . . . , sα}.\". What does equiprobable mean here? What's a region? How does one map x[w] to a choice of s[w]? i.e. what's the function phi? It feels not informative enough to cite another work and not mention what's actually done inline in thetext since it causes the reader to open another paper to understand a key detail. Skipping past it as a reader and just noting to self that there is somehow a tokenization that happens. It's referred to as a \"symbolic regression\" so I assume there is some function phi trained *somehow*.\n\n- sparse self attention\n\n  - Try to avoid introducing a mathematical symbol without defining it. For example, a(m_i, theta_a) is introducdd in line 195 in 3.2.2. but m isnt defined until the next section, which causes the reader extra mental burden in the earlier section.\n  - Again, the \"max-mean sparisty metric\" from Zhou et al [63] used to compute the sparse self attention seems like a key detail but there's not even a basic math expression for it, instead it's delegated to appendix + citation. Like the earlier example with tokenization, this makes the details of the paper hard to follow as a reader must scroll up and down and open other papers to know what's going on.\n\n\nResults\n\n  - the main table 2 includes with and without SAX, and the ablation includes full vs sparse attention (two kinds). Was the ablation done with or without SAX? Also, what does full vs all-full attention mean?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths: \n- good connections with previous work \n- consideration for both tokenization and prediction challenges in multivariate multi-modal time series\n- attention to practical details like missingness\n- attention to computational details like flops imposed by the architectural changes\n- good benchmarking and many baselines run as far as dataset/model combos \n\nWeaknesses:\n- mostly writing clarity. A lot of complicated moving pieces and some key ones are deferred to references and appendices. See \"Questions\" below.", "was_revised": true, "final_justification": "My positive opinion of the work remains unchanged.", "rebuttal": "Thank you for the encouraging review and the constructive feedback on improving our manuscript. We provide our response below :\n\n\n> Provide inline explanation of the symbolic tokenization instead of citing original reference for better readability.\n\nThank you for the suggestion. We will add in the revision the following description of the breakpoint generation and mapping in the main text for easy referencing of the reader :\n\n\"*To ensure an equiprobable distribution of symbols, we follow the empirical observation by Lin et. al [28] that normalized time series sequences follow a Gaussian distribution. We partition this distribution into $\\alpha$ equal-sized areas under the Gaussian curve, referred to as regions, using breakpoints $\\{\\beta_0, \\beta_1, \\ldots, \\beta_\\alpha\\}$, where each region corresponds to a unique symbol. The normalized piecewise aggregated value $\\hat{x}[w]$ is then mapped to a symbolic token $s[w] \\in \\{s_1, \\ldots, s_\\alpha\\}$ based on the breakpoint interval in which it falls, denoted using the mapping function $\\phi(\\hat{x}[w])$.*\"\n\n---\n\n>  \"max-mean sparsity metric\" from Zhou et al [63] used to compute the sparse self attention seems like a key detail but it's delegated to appendix + citation.\n\nThank you. We will move the max-mean sparsity metric details from Appendix B to the Section 3.2.2 to improve readability. \n\n---\n\n> a(m_i, theta_a) is introduced in line 195 in 3.2.2. but m isnt defined until the next section, which causes the reader extra mental burden in the earlier section.\n\n\nWe will reorganize our writing to define $\\mathbf{m}_i$ in Section 3.2.2 and explicitly mention that the design of $a(\\mathbf{m}_i; \\theta_a)$ is detailed in the following Section 3.2.3. Here is the planned updated text from line 195 : \n\n\"*.... with a key modification: the sparsity budget is adaptively controlled by the learned function $a(\\mathbf{m}_i; \\theta_a)$, where $\\mathbf{m}_i \\in \\mathbb{R}^{1 \\times M}$ is a logit vector indicating the presence of each modality for an input sample $s^j_i$. Details on training $a(\\mathbf{m}_i; \\theta_a)$ are provided in Section 3.2.3.*\"\n\n\n---\n> Clarification of the experimental setup in the ablation studies -- Were they done with or without SAX? Difference between full vs. all-full attention.\n\nWe conduct two kinds of design deep-dives in Section 4.2 –\n\n1) Ablation of the design components in Figure 6: The first group of bars shows the ablation without SAX and demonstrates that this tokenization leads to an 8.5% and 5% improvement under full and 40% missingness in modalities during inference, respectively. For the remaining components, we conduct experiments with SAX (as it is a part of MAESTRO unless explicitly mentioned otherwise).\n\n2) Sparse attention-derived Computational efficiency of MAESTRO: All experiments in this section are conducted with SAX. In the first two rows after MAESTRO, ``Full-Attn (Per-Modal)`` and ``Full-Attn (Cross-Modal)`` refer to variants where only the per-modal or cross-modal components, respectively, are replaced with dense attention. ``All Full-Attention`` indicates that both the per-modal and cross-modal attention modules use dense attention mechanisms instead of the proposed sparse design. We will update in the revision the caption of Table 3 as follows : \n\n    \"*Computational Complexity. In MAESTRO, sparse attention in the Per-Modal and Cross-Modal components is replaced by dense attention, referred to as Full-Attn (Per-Modal) and Full-Attn (Cross-Modal), respectively. Replacing all sparse attention components with dense attention is denoted as All Full-Attn*.\""}, {"review_id": "gzT8R8Pa0e", "reviewer": "Reviewer_sZaS", "summary": "This paper introduces MAESTRO, a new method for learning from time series data collected by multiple sensors (modalities), even when some sensors are missing. It uses symbolic tokens to represent data, gives different attention to each sensor based on its importance, and connects different sensors using efficient sparse attention. It also uses a Mixture-of-Experts system to adapt to different combinations of available sensors. MAESTRO is tested on four real-world datasets and outperforms 10 existing methods, showing strong results even when 40% of the sensor data is missing.", "questions": "1. The paper introduces a modality-aware attention budget mechanism to adaptively control sparsity per modality. Could the authors clarify how the gating network G is trained? Is there any regularization or inductive bias to prevent overfitting, especially when modality availability varies widely across samples?\n\n2. The symbolic SAX tokenization is presented as a key component to reduce sequence length and encode missingness. Could the authors provide more quantitative or qualitative insights into why this representation improves performance (e.g., beyond Table 2 and Figure 3)? Would the model still benefit from SAX if trained on high-resolution sequences with sufficient compute?\n\n3. The paper demonstrates robustness under missing modalities, but it is unclear whether MAESTRO can handle asynchronous modalities (e.g., with misaligned sampling rates and no temporal sync). Could the authors discuss how the current design generalizes to this setting, and whether additional temporal alignment strategies would be needed?\n\n4. While MAESTRO uses sparse attention and MoE to improve efficiency, could the authors discuss performance under strict compute budgets (e.g., edge devices)? How does inference latency scale with modality count and sequence length, especially in high-frequency sensor settings?", "limitations": "Yes. The authors have done a commendable job addressing the limitations of their work.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Strengths: (1) The paper presents a high-quality and well-designed framework. MAESTRO integrates multiple components—symbolic tokenization, adaptive attention budgeting, sparse attention, and MoE routing—into a cohesive pipeline that is robust and efficient. (2) The paper is clearly written, with intuitive explanations, detailed methodology, and helpful visualizations (e.g., Figures 2, 4, 5). The ablation studies and complexity comparisons further support the model design. (3) MAESTRO addresses a highly practical and underexplored challenge in multimodal time series learning: handling dynamic and incomplete sensor data. The ability to work under arbitrary modality combinations is highly relevant for real-world sensing systems. (4) The combination of symbolic time-series representation with adaptive attention and sparse MoE for multimodal fusion is novel. \n\nWeaknesses: (1) While the method is well explained overall, the symbolic tokenization step and adaptive attention budgeting could benefit from more intuitive, real-world analogies or examples for broader accessibility. (2) While the combination is novel, many individual components (e.g., SAX, MoE, sparse attention) are adaptations of known ideas; the contribution lies more in system design than theoretical innovation.", "was_revised": true, "final_justification": "Thank authors for the additional explanations and experiments, including those on edge devices. I want to maintain my score.", "rebuttal": "Thank you for your encouraging review and constructive feedback. We appreciate your recognition of the practical significance of our work and the merit in MAESTRO's design for real-world sensing applications. Please find our response to your questions below:\n\n> Could the authors clarify how the gating network 𝐺 is trained?\n\nThe gating network, $G$, is composed of two feed-forward layers, and the output is the sigmoid of the last layer which produces an $M$-dimensional logit vector (where $M$ is the number of modalities). We train this gating network using the cross-entropy loss (given in lines 280-281) with the end-to-end MAESTRO. We do not use any additional regularization loss. During training, we follow a curriculum learning approach (as mentioned in lines 253–257), which increases the modality dropout monotonically over the course of training epochs, as shown in Figure 9 of the Appendix. This training strategy helps prevent overfitting to any specific modality combinations.\n\nAdditionally, in our initial design iterations, we considered a loss term—sparsity loss—where we aimed to minimize the overall attention budget. The sparsity loss was constructed by taking the sum of the $M$-dimensional logit vector. The final loss considered was the cross-entropy loss and the sparsity loss. However, this did not result in stable training without constraining the sparsity loss. In future work, we will consider proactively sparsifying the attention budget based on the available computational resources.\n\n\n> Could the authors provide more quantitative or qualitative insights into why SAX representation improves performance (e.g., beyond Table 2 and Figure 3)? Would the model still benefit from SAX if trained on high-resolution sequences with sufficient compute?\n\n\nThe key advantages of discrete binning achieved using SAX specifically for MAESTRO are: 1) the ability to reserve a symbol to naturally encode missingness, and 2) to reduce sequence length.\n\nAn additional benefit of SAX—particularly for predictive tasks where discriminative patterns across the complete sequence inform task-specific predictions—is that mapping to a fixed vocabulary helps suppress local noise or spurious perturbations. This allows the model to focus on high-level trends that are more relevant to the task. Many previous works [1, 2, 3] have reaped this advantage and reported improved classification performance.\n\nIn our case, to demonstrate the benefit of SAX tokenization, we conduct following two experiments.\n\n**Experiment I**: Using the WESAD dataset (stress detection with 5 modalities and 3 classes), we trained on high-resolution data without reducing sequence length by setting the word length (w) to 1. The results presented below confirm the benefit of SAX representation.\n\n\n| Tokenization Method       | Sequence Length <br>(after tokenization) | Accuracy | Std. Dev |\n|---------------------------|------------------------------------------|----------|----------|\n| SAX, w=2        | 128                                      | 0.77     | 0.02     |\n| SAX, w=1        | 256                                      | 0.75     | 0.04     |\n|Raw input (No SAX)                    | 256                                      | 0.69     | 0.02     |\n\n\n**Experiment II**: We add small Gaussian noise to 20% of the modalities (in this case, the wrist BVP, TEMP, and EDA) and conduct zero-shot inference. As expected, when the word length is set to 2, the effect of this small perturbation is neglible. Even with word length 1, the performance is much more robust compared to raw input  \n\n| Tokenization Method | Clean Data | 20% Noisy Channels |\n|---------------------|------------|--------------------|\n| SAX, w = 2          | 0.77       | 0.75               |\n| SAX, w = 1          | 0.75       | 0.68               |\n| Raw input (No SAX)  | 0.70       | 0.51               |\n\nWe will add these experiments to our Appendix section and refer to them in Section 3.2.1 to give our readers more intuition about SAX-based tokenization. Thank you for asking about this aspect.\n\nWe would also like to draw your attention to Figure 5 and Section E.1 in the Appendix, which demonstrate that under arbitrary missingness, the proposed MAESTRO with SAX encoding outperforms its counterpart without SAX encoding.\n\n[1] Symbolic representation for time series, 2024\n\n[2] Bake off redux: a review and experimental evaluation of recent time series classification algorithms, 2024\n\n[3] SAX-VSM: Interpretable Time Series Classification Using SAX and Vector Space Model, 2013\n\n> Could the authors discuss how the current design generalizes to the settings of asynchronous modalities (e.g., with misaligned sampling rates and no temporal sync), and whether additional temporal alignment strategies would be needed?\n\nIn the current work, we focus on handling complete modality missingness, which is a common issue in real-world sensing applications with a large number of modalities (N > 4). This typically occurs when a sensor fails and we are left with an arbitrary subset of multimodal inputs for prediction, as described in our paper's scope (lines 10, 62, 74).\n\nWhile a comprehensive study evaluating MAESTRO on irregularly sampled, asynchronous time-series is out of the scope of the current paper, we present a brief pilot study in a simplified setting. We mask approximately 25% of the samples from the beginning and end of the sequence in 20% of the modalities(in this case, the wrist BVP, TEMP, and EDA) during inference to demonstrate zero-shot transfer capability. Currently, the model treats this partial missingness similarly or slightly better than complete missingness. \n\n\n| Modalities                                      | Accuracy |\n|------------------------------------------------|----------|\n| All available                                  | 0.77     |\n| Completely missing 20% modalities              | 0.61     |\n| Mask first 25% time steps in 20% modalities    | 0.60     |\n| Mask last 25% time steps in 20% modalities     | 0.65     |\n\n\nThis performance can potentially be improved by incorporating standard techniques such as absolute positional encoding, or employing prior models like mTAN [4] (multi-time attention network), which can better encode irregular time-series with semantic meaning to support downstream cross-modal learning. We will include this discussion and pilot experiment in our Appendix, and plan to explore these directions in the future. Thank you for commenting on this.\n\n[4] Multi-Time Attention Networks for Irregularly Sampled Time Series, 2021\n\n\n> Could the authors discuss performance under strict compute budgets (e.g., edge devices)? How does inference latency scale with modality count and sequence length, especially in high-frequency sensor settings?\n\nWe conduct two experiments to demonstrate 1) MAESTRO's performance across two computation platforms and 2) benefit of sparse attention with simulated high sampling rate data stream.\n\n**Experiment I**: We implement MAESTRO with different number of experts and varying word lengths (which in turn reduce the sequence length), and report the accuracy, GFLOPs, and latency on two compute platforms—RTX A6000 and Jetson TX2.\n\n##### Varying Experts for MoE (default w = 2)\n\n\n| # Experts | ACC_eval | RTX A6000 - Latency (ms) | RTX A6000 - GFLOPs  | Jetson TX2 - Latency (ms)   | Jetson TX2 - GFLOPs |\n|-----------|----------|--------------------------|---------------------|-----------------------------|---------------------|\n| 8\t\t    |   0.80   |     352.09\t              |       7.15          |\t 1298.52\t              |      6.16           |\n| 4\t\t    |   0.79   |     279.41\t              |       7.08          |\t 1315.08\t              |      6.12           |\n| 2\t\t    |   0.75   |     245.03\t              |       7.07          |\t 1291.25\t              |      6              |\n\n\n##### Varying Word Length for SAX (default w = 2)\n\n| Word Length (w) | ACC_eval | RTX A6000 - Latency (ms) | RTX A6000 - GFLOPs | Jetson TX2 - Latency (ms) | Jetson TX2 - GFLOPs |\n|------------------|----------|---------------------------|----------------------|-----------------------------|-----------------------|\n|2\t\t           |  0.80    |  279.41                   |\t  7.08\t             |        1315.08\t            |       6.12           |\n|4\t\t           |  0.77    |  270.00                   |\t  3.56\t             |        677.81\t             |      3.06           |\n|8\t\t           |  0.80    |  275.69                   |\t  1.74\t             |        349.33\t             |      1.5           |\n\n\n**Experiment II**: We resample the modalities in WESAD to increase the sequence length in order to emulate high-frequency sensor readings and report the performance below. As expected, our proposed MAESTRO performs consistently even with longer sequence lengths. However, upon replacing the sparse attention with canonical dense self-attention layers in both the intra-modal and cross-modal stages, we encounter out-of-memory (OOM) issues at higher sampling rates (in this case, 128 Hz), highlighting MAESTRO’s advantage in resource efficiency. All experiments were conducted on the RTX A6000.\n\n\n| Sampling Rate | MAESTRO - Acc | MAESTRO - GFLOPs | MAESTRO - MMACs | Dense Attention - Acc | Dense Attention - GFLOPs | Dense Attention - MMACs |\n|---------------|----------------|-------------------|------------------|-------------------------|---------------------------|--------------------------|\n| 32            | 0.77           | 6.13              | 3066             | 0.75                    | 8.78                      | 4205                     |\n| 64            | 0.77           | 14.41             | 7205             | 0.71                    | 23.02                     | 11510                    |\n| 128           | 0.73           | 29.01             | 14502            | OOM                     | OOM                       | OOM                      |"}, {"review_id": "y9JTfnZtJ6", "reviewer": "Reviewer_bCcz", "summary": "MAESTRO is a multimodal learning framework designed for dynamic time-series data from heterogeneous sensors, addressing challenges like missing modalities and inefficient pairwise modeling. It uses symbolic tokenization to represent time-series data compactly while explicitly marking missing inputs. An adaptive sparse attention mechanism allocates attention based on modality relevance, and a sparse mixture-of-experts layer enables dynamic routing depending on available modalities. Experiments on four real-world datasets show that MAESTRO outperforms state-of-the-art baselines in both accuracy and robustness, especially under partial modality settings.", "questions": "1. The framework is only evaluated on classification problems. What about its applicability to other tasks of different type?\n2. Although the proposed method handles missing modalities well, the model's behavior under noisy inputs is not systematically evaluated. I am curious about its robustness to the noisy data.\n3. One minor issue: \"reliance on\" in line 5 should be put ahead of \"(1)\"?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- MAESTRO is robust to missing modalities. It handles arbitrary combinations of available sensor modalities using symbolic tokenization and dynamic routing, outperforming baselines.\n- By employing sparse attention and avoiding pairwise interaction modeling, MAESTRO significantly reduces computational cost compared to dense multimodal methods like MULT.\n\nWeaknesses\n\n- It seems that the model involves multiple components that may require extensive hyperparameter tuning for the new datasets.", "was_revised": true, "final_justification": "The authors have addressed my concerns. I have no more concern.", "rebuttal": "Thank you for the encouraging review and the recognition of MAESTRO's performance and efficiency under practical settings of learning from heterogenous time-series modalities. We provide our response below : \n\n> It seems that the model involves multiple components that may require extensive hyperparameter tuning for the new datasets.\n\nIn our current work we have not extensively fine-tuned our hyper-parameters ( (as shown in Table 5 of the Appendix), yet we are able to outperform the existing baselines under complete and arbitrary modality combinations. However, for application-specific scenarios, based on the available computational resources, these hyperparameters can be selected to optimize for performance and computational efficiency. We present a brief sensitivity analyses for number of experts and the word length for the WESAD dataset across two computational platforms below :\n\n##### Varying Experts for MoE (default w = 2)\n\n\n| # Experts | ACC | RTX A6000 - Latency (ms) | RTX A6000 - GFLOPs  | Jetson TX2 - Latency (ms)   | Jetson TX2 - GFLOPs |\n|-----------|----------|--------------------------|---------------------|-----------------------------|---------------------|\n| 8\t\t    |   0.80   |     352.09\t              |       7.15          |\t 1298.52\t              |      6.16           |\n| 4\t\t    |   0.79   |     279.41\t              |       7.08          |\t 1315.08\t              |      6.12           |\n| 2\t\t    |   0.75   |     245.03\t              |       7.07          |\t 1291.25\t              |      6              |\n\n\n##### Varying Word Length for SAX (default w = 2)\n\n| Word Length (w) | ACC | RTX A6000 - Latency (ms) | RTX A6000 - GFLOPs | Jetson TX2 - Latency (ms) | Jetson TX2 - GFLOPs |\n|------------------|----------|---------------------------|----------------------|-----------------------------|-----------------------|\n|2\t\t           |  0.80    |  279.41                   |\t  7.08\t             |        1315.08\t            |       6.12           |\n|4\t\t           |  0.77    |  270.00                   |\t  3.56\t             |        677.81\t             |      3.06           |\n|8\t\t           |  0.80    |  275.69                   |\t  1.74\t             |        349.33\t             |      1.5           |\n\n\nThank you for commenting on this aspect. We will include this analyses in our Appendix.\n\n\n\n> MAESTRO is evaluated on classification tasks; discuss applicability to other tasks.\n\nWe currently explore classification tasks as they are the commonly occuring multimodal tasks (activity recognition, clinical disease identification, stress detection etc.) in consumer and clinical healthcare and lifestyle applications involving multiple heterogenous sensors collecting time-series data, which is the focus of our work. Also current multimodal methods with time-series are designed for predictive applications allowing us to benchmark MAESTRO.\n\nWe think, fundamentally MAESTRO's arechitecture can be extended to support generative tasks with mulitmodal input. In future work, we plan to explore two key directions for such extension: 1) curating a multimodal generative tasks dataset; and 2) exploring encoder-only vs. encoder-decoder cross-attention architectures for generation. One of the generative application spaces that stands to benefit from cross-modal attention to heterogeneous modalities without resource efficiency is robot manipulation. Recent works [1, 2] in robot manipulation explore enriching the latent representations of the action sequence through multimodal inputs before learning the Reinforcement-Learning policy. Often, there are time-series modalities [3, 4] that are availabile arbitrarily in these applications. We plan to explore the applicability of MAESTRO to such settings (to learn better action-sequence representation from multimodal inputs) in future.\n\nWe will add this discussion to our \"Broader Impact and Future Work\".\n\n\n[1] Decision Transformer: Reinforcement Learning via Sequence Modeling, 2021 \n\n[2] A Vision-Language-Action Flow Model for General Robot Control, 2025\n\n[3] MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation, 2024\n\n[4] The MOTIF Hand: A Robotic Hand for Multimodal Observations with Thermal, Inertial, and Force Sensors, 2025\n\n---\n\n> While MAESTRO handles missing data well, the model's behavior under noisy inputs is not systematically evaluated. I am curious about its robustness to the noisy data.\n\nIn the current work, we are focused on handling complete modality missingness, which is a common issue in real-world sensing applications with a large number of modalities (N > 4), where a sensor fails and we are left with an arbitrary set of multimodal inputs to make predictions, as mentioned in our paper's scope (lines 10, 62, 74).\n\nWhile an extensive study on noise robustness is out of scope for this paper, which focuses on robustness to missing modalities, it could be a promising direction for future work on MAESTRO. We have just conducted some initial exploration with three types of noise defined as follows. Here $\\tilde{\\mathbf{x}}$ denotes the subset of modalities chosen for introducing corruption or omission :\n\n1. Random noise : Selected channels are replaced with i.i.d. Gaussian noise, $\\tilde{\\mathbf{x}} = \\boldsymbol{\\epsilon}, \\quad \\boldsymbol{\\epsilon} \\sim \\mathcal{N}(0, \\sigma^2), \\quad \\text{for } c \\in \\mathcal{C}$\n2. Additive noise  :  Gaussian noise is added to the original signal, $\\tilde{\\mathbf{x}} = \\mathbf{x} + \\boldsymbol{\\epsilon}, \\quad \\boldsymbol{\\epsilon} \\sim \\mathcal{N}(0, \\sigma^2), \\quad \\text{for } c \\in \\mathcal{C}$.\n3. Additive noise with spikes (representing electrical spikes) : Sparse high-magnitude impulses (spikes) are added in addition to Gaussian noise: $\\tilde{\\mathbf{x}} = \\mathbf{x} + \\boldsymbol{\\epsilon} + \\mathbf{M} \\odot \\mathbf{S}, \\quad \\boldsymbol{\\epsilon} \\sim \\mathcal{N}(0, \\sigma^2)$,\n   where $\\mathbf{M} \\sim \\text{Bernoulli}(p)$ is a binary mask indicating spike locations  with $p$ probability, $\\mathbf{S} \\in \\{-m, +m\\}$ is the spike magnitude, $\\odot$ denotes element-wise multiplication  \n\n\n\nWe present the zero-shot performance of MAESTRO on noisy WESAD data in the following two combinations : in combination 1, we dropped the chest accelerometer, and in combination 2, we dropped the wrist BVP, EDA, and Temp. Please refer to Table 6 in our Appendix for more details on the WESAD modalities.\n\n| Corruption Type            | Combination 1              | Combination 2                       |\n|----------------------------|---------------------------|-----------------------------------|\n| *None (Clean and Complete Modalities)*               | *0.77*                      | *0.77*                              |\n| Missing                    | 0.61                      | 0.65                              |\n| Random Noise               | 0.61                      | 0.63                              |\n| Additive Noise (σ = 0.5)   | 0.75                      | 0.74                              |\n| Additive Noise + Spikes (σ = 0.5, p = 0.1, m = 5)   | 0.60                      | 0.73                              |\n\nWe observe that for simple noise settings like random noise, MAESTRO treats those modalities similar to complete missingness of that modality.\n\nIn the simple additive noise setting, we are able to perform almost as well as the case where all modalities are present. This is not surprising because (1) our SAX representation provides robustness to local perturbation, and (2) our cross-modal attention allows querying only for the relevant inter- and intra-modal values relevant to the task.\n\nFinally, a more aggressive noise where we simulate electrical noise interference as spikes shows the same performance in one combination as missingness and in another combination very close to the full modality scenario.\n\nWe can consider missingness to be one of the aggressive noise corruptions, and through this controlled experiment, we can see MAESTRO's robustness to simple perturbations. Thank you for raising this aspect of MAESTRO's robustness to noisy inputs. We will include the above preliminary analyses in our Appendix to motivate and guide potential future exploration, e.g., to explore other systematic noise injection mechanisms and their impact on MAESTRO.\n\n---\n\n> Minor writing issue: \"reliance on\" in line 5 should be put ahead of \"(1)\"?\n\nWe will update our abstract to reflect the suggested flow."}]

Author rebuttal:

Meta-review: # Summary
This paper proposes MAESTRO, a novel approach for learning from multimodal time series with missing modalities.

# Strength
* The proposed method is robust to missing modalities.
* Strong improvements over state of the art for multiple datasets.
* A well-designed framework, which integrates multiple components into a cohesive pipeline.

# Weakness
* Limited novelty with ad-hoc algorithms

# Recommendation
AC reads all reviews and agrees that (i) the technical contributions of proposed method are great; and (ii) the solving problem is a highly practical and under-explored challenge in multimodal time series modeling and it will be useful in various applications. Therefore, AC recommends an acceptance.
Decision: Accept (spotlight)