MIAM: Modality Imbalance-Aware Masking for Multimodal Ecological Applications
Authors: Robin Zbinden
Wesley Monteith-Finas
Gencer Sumbul
Nina van Tiel
Chiara Vanalli
Devis Tuia
Venue: ICLR 2026 Poster (2026)
Keywords: multimodality
masking
modality imbalance
ecology
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: MIAM: a masking strategy to address modality imbalance in the context of multimodal ecological applications

Abstract: Multimodal learning is crucial for ecological applications, which rely on heterogeneous data sources (e.g., satellite imagery, environmental time series, tabular predictors, bioacoustics) but often suffer from incomplete data across and within modalities (e.g., unavailable satellite image due to cloud cover, missing records in a time series). While data masking strategies have been used to improve robustness to missing data by exposing models to varying input subsets during training, existing approaches typically rely on static masking and inadequately explore the space of input combinations. As a result, they fail to address modality imbalance, a critical challenge in multimodal learning where dominant modalities hinder the optimization of others. To fill this gap, we introduce Modality Imbalance-Aware Masking (MIAM), a dynamic masking strategy that: (i) explores the full space of input combinations; (ii) prioritizes informative or challenging subsets; and (iii) adaptively increases the masking probability of dominant modalities based on their relative performance and learning dynamics. We evaluate MIAM on two key ecological datasets, GeoPlant and TaxaBench, with diverse modality configurations, and show that MIAM significantly improves robustness and predictive performance over previous masking strategies. In addition, MIAM supports fine-grained contribution analysis across and within modalities, revealing which variables, time segments, or image regions most strongly drive performance.

Published as a conference paper at ICLR 2026
MIAM: MODALITY IMBALANCE-AWARE MASKING FOR MULTIMODAL ECOLOGICAL APPLICATIONS
Robin Zbinden; Wesley Monteith-Finas; Gencer Sumbul, Nina van Tiel,
Chiara Vanalli, Devis Tuia
École Polytechnique Fédérale de Lausanne (EPFL), Switzerland
ABSTRACT
Multimodal learning is crucial for ecological applications, which rely on hetero- geneous data sources (e.g., satellite imagery, environmental time series, tabular predictors, bioacoustics) but often suffer from incomplete data across and within modalities (e.g., unavailable satellite image due to cloud cover, missing records in a time series). While data masking strategies have been used to improve robust- ness missing data by exposing models varying input subsets during training, existing approaches typically rely on static masking and inadequately explore the space of input combinations. As a result, they fail to address modality imbalance, a critical challenge in multimodal learning where dominant modalities hinder the optimization of others. To fill this gap, we introduce Modality Imbalance- Aware Masking (MIAM), dynamic masking strategy that: (i) explores the full space of input combinations; (ii) prioritizes informative or challenging subsets; and (iii) adaptively increases the masking probability of dominant modalities based on their relative performance and learning dynamics. We evaluate MIAM on two key ecological datasets, GeoPlant and TaxaBench, with diverse modality config- urations, and show that MIAM significantly improves robustness and predictive performance over previous masking strategies. In addition. MIAM supports fine- grained contribution analysis across and within modalities, revealing which vari- ables, time segments, or image regions most strongly drive performance.
INTRODUCTION
Ecological modeling plays a central role in conservation, climate change adaptation, and environ- mental management (Pollock et al. 2025). Capturing complex ecological processes requires data that reflect multiple facets of both the environment and the species of interest. Consequently, eco- logical datasets are inherently multimodal (Hartig et al. (0244/Maa et al., (025),, integrating diverse inputs such as tabular environmental variables (e.g., elevation, soil properties), time series (e.g., cli- mate records), audio (e.g., bioacoustics), natural images (e.g., species observations), and satellite imagery (Picek et al. 2024 Sastry et al.) 2025). Learning effectively from this heterogeneous data presents several challenges. First, ecological data are frequently incomplete due to limitations in data collection, such cloud-obstructed satellite images or temporal gaps in monitoring efforts. Missing data can occur at the modality level (e.g., no image available for a location) or within modalities (e.g., missing entries in climate time series). Second, quantifying the importance of different in- puts is critical, as these models also aim to provide ecological insights. This includes contributions both across modalities (e.g., how useful is satellite imagery vs. tabular data?) and within them (e.g.. which year in a time series matters the most?). Addressing these challenges requires models that can flexibly operate on arbitrary and incomplete subsets of inputs.
Recent advances in multimodal learning have made progress toward this goal through data masking, where a masking strategy specifies a probability distribution over which inputs are hidden from the model during training. Models such as 4M (Mizrahi et al. 2023 Bachmann et al. 2024) implement this idea by randomly masking subsets of inputs, simulating missing data. This exposes the model to diverse modalities and feature combinations, promoting robustness to incomplete inputs and en- abling contribution techniques such as Shapley-based feature importance (Zbinden al. 2026
"Equal contribution. Correspondence to: robi www.ziina @bbinen@@ep1.ch

---

Published as a conference paper at ICLR 2026
However, these masking distributions do not adequately explore the space of imput subsets and, be- ing typically fixed and uniform, do not adapt to evolving learning dynamics or modality specific characteristics during training.
As a result, such approaches do not address
Validation AUC (%)
Unimodal (aracle)
crucial challenge in multimodal learning: modality imbalance (also known as modality
MIAM (ours)
Modality dropout
competition), This occurs when some modal- ities dominate the learning process, captur- ing most of the predictive signal and gradi- ent flow, thereby impeding the optimization of other, potentially complementary, modali- ties (Wang et al., 2020) Huang et al. [2022] Wu jt 2002). An example of modality imbal- ance is shown in Fig.
Uniform
Constant
Dirichlet
60
10
20
30
40
50
Epoch
Several mitigation strategies have been proposed, including gradi- ent reweighting (Peng et al.) 2022). knowledge distillation from unimodal teachers (Du et al. 2021], and adaptive training schedules based on per-modality learning speeds (Wu et al.] 2022). However, these methods often require additional components or supervision, while even simple modality dropout (Neverova et al. 2013) has proven to be competitive with more complex approaches. Building on this,Weii al. (2024) proposed On-the-fly Prediction Modulation (OPM), masking strategy that adjusts per-modality probabilities based on relative performance scores. However, these scores remain nearly static during training, operate only at the modality level (masking entire modality or none of it), and fail to fully explore the space of possible input combinations.
Figure 1: Modality imbalance. On the GeoPlant dataset, multimodal masking-based approaches that ignore modality imbalance underperform unimodal model when evaluated using satellite imagery only, as dominant modalities hinder ef- fective optimization. MIAM closes this gap by adaptively increasing the masking probability of dominant modalities based their relative per- formance and learning dynamics.
In this work, we propose Multimodal Imbalance-Aweer Aware Masking (MIAM), a principled, dynamic, score-driven masking strategy illustrated in Fig. We first formalize masking strategies as probabil- ity distributions unit hypercubes and identify three key properties often missing from existing approaches. An effective masking strategy should have full support on the hypercube, prioritize corners while assigning higher weight to those corresponding to modality imbalance by adjusting masking probabilities based on modality dominance. These in- sights motivate MIAM. which is designed to: i) handle arbitrary missing inputs; ii) mitigate modal- ity imbalance by adjusting masking of dominant modalities; and iii) support both within and across modality contributions. To achieve this, MIAM constructs a mixture of product beta distributions to define masking probabilities and dynamically adjusts this distribution during training based on modality-specific performance and learning speed. We evaluate MIAM on two ecological bench- marks: GeoPlant (Picek et al., (024) for species distribution modeling and TaxaBench (Sastry al.) 2025) for multimodal species classification, spanning three and five modalities, respectively. MIAM consistently outperforms existing masking approaches, with particularly strong gains for modalities affected by modality imbalance. Beyond predictive performance, MIAM also provides ecological insight by revealing
key input configurations, and adapt
only which modalities the most influential, but also which predictors, temporal segments, or image regions drive model performance highlighting key ecological sig- nals such NDVI and heatwaves. Together, these results underscore the importance of principled masking in multimodal learning, and particularly so in ecological applications where data are het- erogeneous and incomplete. Code available atthtps: /ttpbbgggttuuuu//// con/ http//bbrrob/iMAA.
2 RELATED WORK
Multimodal learning aims at integrating information from heterogeneous data sources such as au-
dio, image, video, text, and tabular data (Baltrušaitis et al.) 2018 Uppal et al. 2022 Xu et al. 2023 Zhang et al. 2023, Bachmann et al. 2024 Zong et (025).. deally. different modalities provide synergistic views of the underlying process (Dffumier al. (2025_ ) ecology, where multimodal
approaches are increasingly adopted to leverage the diversity of available data sources (Miao et al.
2025 Hartig al. 2024
Picek et al. 2024). However, combining multiple modalities often
2

---

Published as a conference paper at ICLR 2026
Pt. P2. P3
MixProdBeta (Pam))
Performance
Sm
M
Modality speliit---eciiicc Tokenization
Prediction
dm = 5
Fusion Prediction head
0.0
Tabular data
MASK
0.7
Epoch
MASK
41.9
0.0
t, d1
1
Pa
0.2
Satellite Image
S2 f. d2
P2
Ps
S3 1 d3
P3 +
(a)
(b)
Figure 2: Overview of MIAM. (a) Each token of modality m is masked with probability pms sampled from mixture of product beta distributions. (b) The distribution parameters modulated by Pam.. and Pd... derived from the per-modality performance 8m and its absolute derivative d., to address modality imbalance. Modalities with relatively high sm and low dm masked more often.
troduces the challenge of modality imbalance, where dominant modalities impede the optimization of less informative ones. This issue can arise from differences in predictive strength, input scale, learning speed (Wang al. 2020; |eng et al., 2022 Huang al., 2022). Wang al. (2020) shows that multimodal models can underperform their unimodal counterparts due to differing gen- eralization rates across modalities, and proposed gradient blending address this effect. Building on this, Peng et al. (2022) introduces On-the-fly Gradient Modulation (OGM) to adjust gradients based the discrepancy of modalities' contributions, while Wu et al. (202) proposed adapting learning speeds to counteract greedy optimization be- haviors. Other approaches adjust training using unimodal teacher distillation Du al. (2021) or prototype-based regularization (Fan et al. 2023). While effective when all modalities are present, these methods assume complete inputs and are not designed to cope with missing modalities or arbitrary subsets conditions that are especially common in ecological datasets.
training schedules based on modality-specinn
Masking is widely used in self-supervised learning (SSL) as a pretext task to learn robust, general- purpose representations. By reconstructing masked inputs from partial context, models can learn without labels and develop deeper contextual understanding (Devlin et al. 2019: He et al. 2022). Beyond SSL, masking also promotes robustness missing inputs. In supervised settings, modal- ity dropout (Neverova et al. 2015) randomly masking modalities during training has shown competitive performance for handling missing modalities. MultiMAE (Bachmann al.) 2022)
and 4M (Mizrahi et al.] 20233 Bachmann et al., 2024) extend this idea by masking and recon- structing both across and within modalities, enabling models flexibly handle arbitrary subsets of inputs while also supporting per-modality performance analysis. Similarly, [overt et al. (2023) leverages masking estimate image patch contributions through Shapley values. In ecology, MaskSDM (Zbinden et al. 2026) applies uniform random masking to tabular predictors and satel- lite embeddings. improving robustness to missing data both across and within modalities. another direction, Wei et al. (2024) adapts masking to address modality imbalance by adjusting dropout probabilities based on per-modality performance scores. Despite their success, most of the exist- ing masking strategies rely uniform distributions over limited subset of input combinations. overlooking modality imbalance and lacking robustness to arbitrary missing inputs. In Section 3.1 formalize existing masking strategies, identify their limitations, and propose key principles for effective masking in multimodal settings that serve as the foundation for MIAM.
3 METHODOLOGY
3.1 PROBLEM SETUP AND MASKING FORMULATION
In our multimodal setup, each input sample r consists of modalities, represented as a tu- ple x2 ,..., M), where each modality xm is associated with T., tokens: x.
3

---

Published as a conference paper at ICLR 2026
Pj
P3
(a) Constant
(b) Uniform
(c) Dirichlet
(d) Modality dropout
0
(e) OPM
(f) Uniform hypercube
(g) Beta hypercube
(h) MIAM
Figure 3: Masking strategies viewed as probability distributions p (P1: (PP1-P2: P2. P3) over the unit hypereube, illustrated with three modalities where modality is dominated. 5000 points are drawn per strategy visualize the distribution. (a) Constant masking (e.g.. 0.75) (b) Shared probabil- ity p 4(0 1) across modalities (Zbinden et al. 2026) (c) Symmetrie Dirichlet with a = 1M (Mizrahi al. 2023). (d) Modality dropout with fixed probability (Neverova et al. 2015) (e) OPM (dynamic masking): modality is never masked (Wei et al. 2024) (f) Uniform hypereube: inde- pendent Pm (000 per modality for full support. (g) Beta hypercube: mixture of product beta distributions to prioritize corners (h) MIAM (dynamic masking): beta hypercube with imbalance-
aware adjustments, causing modality be masked less often than the others.
22 Each token is a sotb-iimensonall vector obtained through tokenizer and en- codes distinct, non-overlapping information within its modality. These tokens are then fused to pro- duce the final prediction, here using a transformer architecture (Vaswani al. 2017] Mizrahi al.) assume that all tokens within a given modality share the same masking prob- ability, denoted P.. Collectively, these form the masking probability vector = (P1 P2...., .......Pa. P&r) e [0, 1] which lies in the M-dimensional unit hypercube. A masking strategy is then defined probability distribution over this hypercube, which may evolve during training.
(ITT) 22 2m
2023). Importantly,
Existing masking strategies. The simplest strategy assigns fixed masking probability all modali- ties, e.g., constant P... 0.75 (He et al. 2022). To introduce variability in the number of observed tokens, MaskSDM (Zbinden al. 2026) samples a single masking probability ~ 2d(0, 1) per batch and applies it uniformly: p p- IM- However, when modalities contain multiple tokens, the chance of observing only one modality mn' (i.e., all others being fully masked) scales as mmmmpmm PTT". which decreases exponentially with the number of tokens. Thus, it is highly probable that the model is exposed to at least one token from a dominant modality, making it difficult to learn from other modalities. To encourage more structured input variations. 4M (Mizrahi al. 2023) samples p from a symmetric Dirichlet distribution: ~ Dir(ax) with a = lus, resulting in uniform sampling over the simplex. However, this constrains the expected proportion of visible tokens to approxi- mately 1/M at each iteration. thereby limiting the diversity of the observed input configurations. Modality dropout (Neverova et al., 2015)) masks each modality with fixed probability and, like 4M, treats all modalities equally thereby ignoring modality competition. OPM (Wei et al. 2024) builds on modality dropout by masking more discriminative modalíties more frequently. However, OPM restricts p E {0, 1 each modality is either fully masked or fully visible preventing partial masking and limiting fine- imningrined contribution. Moreover, modalities with low performance scores never masked (p... 0), while others masked at near-constant rates (see Appendix A.3) For clarity, the masking strategies discussed above are illustrated in Fig. using a 3-dimensional hypereube (i.e., three modalities).
4

---

Published as a conference paper at ICLR 2026
Design principles for effective masking. We identify three key properties that effective masking strategy over p E [0, 1] should satisfy. (i) Full support: the distribution should assign non-zero probability to every p. ensuring that any combination of masked and unmasked tokens occur. (ii) Corner prioritization: points near the corners of the hypercube should be sampled more often, ensuring that the model frequently observes combinations with either almost all tokens almost none from each modality. In ecological settings, incomplete data often occurs at the modality level rather than at the token level, making full-presence and full-absence cases especially important. In addition, the corners corresponding to (0. (0.0..... 0) and (1, 1.......1 1) should be prioritized, as they expose the model to scenarios where either all modalities are available or only few tokens remain. This helps the model remain reliable both when all modalities are present and when fine- grained contribution analyses require operating with very few tokens. Further motivation for corner prioritization is provided in Appendix A.2.2 (iii) Imbalance-awarenee
the masking distribution should explicitly address modality imbalance by assigning higher masking probabilities to dominant modalities, which can be identified based on metrics like modality performance or learning speed.
3.2 MIAM: MODALITY IMBALANCE-AWARY MASKING
We introduce MIAM, masking strategy designed to satisfy the three key principles outlined above. First, to ensure full support, we consider the uniform hypercube distribution, sampling p uniformly over the hypercube [0, 11MM, i.e., drawing each Pm (0, 1) independently. Second, to prioritize the corners of the hypercube, we construct a mixture product beta distributions, each concentrat- ing probability mass around different corner of the hypercube. This formulation also allows to increase the likelihood of sampling near the two key corners (0, O.... 0) and (1, 1.... ......................... 1). which call the beta hypercube distribution. Finally, MIAM addresses modality imbalance by dynami- cally adjusting the parameters of the beta hypercube distribution based on modality-specife dynamics. We identify dominant modalities by jointly considering their relative performance and learning speed, with the latter estimated as the temporal derivative of performance during train- ing. These modalities are then masked more frequently, encouraging the model to better leverage under- mnder-optimized inputs.
learning
Corner-anchored mixture of product beta distributions. To construct flexible, con-uniform probability distribution over the hypercube [0, 1]M we define a mixture of product beta distributions, where each mixture component is designed to concentrate most of its probability mass near one of the M corners. Let Beta(p,,. (x, denote the beta probability density function evaluated at Pm
(0, 1). For given corner c = (cI,..., ... CM) ) E {0, 13M and input p = .P......PPs! E [0, 1JM, we define the product beta distribution anchored at corner e as:
M [eeta(pm: mi 1, 1, if cm = 0, Beta N. if Cm = 1,
fc(p)
II m==1
where the sharpness parameter k > controls the concentration around the corner. We then define the mixture distribution over the set of corners C = 1)
MixProdBeta(p) = Σ We fo(p),
cEC
where the weights {we)eec are nonnegative and sum to 1. We allow asymmetric weighting to emphasize specific corners for example, assigning larger weights to the corners (0...., 0) and (I,...,1) 1). Specifically, we set
if {(O,....0), (I...., 1)), otherwise.
We
2
This weighting allocates half of the mass to the two selected corners and evenly distributes the rest between the remaining 2d 2 corners. In the particular case when each modality has only one token, prioritizing corner (1...., 1) is meaningless, as masks all inputs. In this case, we reassign its weight to corner (0, (O.....00 0), yielding W(o.....)) Y while leaving the other weights unchanged.
Modality imbalance coefficients. To mitigate modality imbalance during training, we modulate the sharpness parameter of the comner-anchored beta distribution using two modality-specific factors
5

---

Published as a conference paper at ICLR 2026
Pe.. and Pdm- The coefficient Pem is computed from Sm. the performance score for modality m when evaluated in isolation on a chosen validation set and metric. In contrast, Pd... is calculated from the absolute derivative dm of S.n Both coefficients are normalized via the geometric mean across modalities:
dm
Pem
[mm== Sm) I/M'
Pdm
1/M
(4)
M (mm== dmm)
A high Psm. indicates that modality m achieves strong unimodal performance, whereas a high pam. reflects rapid improvement decline in performance. The ratio Ps../ thus guides masking: modalities with high and stable performance (high ratio) are masked more frequently, allowing the model focus on modalities that are less performant or still learning, while continuing to explore all input combinations during training" " We incorporate this adaptive masking into the corner-anchored beta distributions by adjusting k asymmetrically, depending on the corner vector c:
Beta
Pmi 1, ni
Parm Palm
if Cm = O.
fc(P) =
II
(5)
A
Beta
Pmn N-
Peess Pdm
if Cm = 1.
Here, À >> O controls the influence of the imbalance ratio Pam on the sharpness adjustment. Modali-
ties with higher ratios produce beta distributions more concentrated near 1, increasing the probability of their masking. For intuition, the marginal distribution of MIAM is shown in Appendix(A.2) A.2.1
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
eco- We evaluate MIAM'S ability handle incomplete data within and across modalities logical datasets with diverse modality types and configurations: GeoPlant (Picck et al. (024) for species distribution modeling (SDM) and TaxaBench (Sastry al., 2025) for multimodal species classification. SDM is cornerstone ecological task that relates species occurrence records to envi- ronmental variables (Elith & Leathwick 2009). where robustness to missing inputs and fine- -rainndd interpretability are essential (Zbinden et al. 2026). Increasingly. these occurrence records span het- erogeneous and often incomplete modalities (e.g., image, audio, geolocation), highlighting the need for models capable of robust multimodal species classification, as evaluated by TaxaBench. A brief overview of each dataset is given here, with full details in Appendix A..
GeoPlant (Picek et al. 2024) integrates three modalities: tabular environmental predictors, Sentinel- 2 satellite imagery, and time series from both climate and Landsat satellite data. The task is to predict the presence absence of plant species at locations across Europe, formulated as a multi-label clas- sification problem. We use the provided vegetation plot survey labels (presence-absence data) and split the data into training (70%), validation (15%), and test (15%) sets using spatial block cross- validation (Roberts et al. 2017). To mitigate spatial autocorrelation, we employ large block sizes (1"x19). We retain only species with more than 20 observations and evaluate those with least one presence record in all three splits, yielding 1783 species. Model performance is assessed using the Area Under the ROC Curve (AUC) averaged across species, the standard metric in SDM. For methods requiring per-modality performance scores (OPM and MIAM), we compute the validation and x 10 for MIAM. We tokenize the satellite images at both the patch level (5x5 patches) and the channel level (Red, Green, Blue, NIR) level; the time series at both the year (2000-2018 for climate: 2000-2020 for Landsat) and channel level (4 climatic variables; Landsat bands); and assign one token to each of the 48 tabular variables. The tokenization linearly encodes image patches and time series segments with positional embeddings (Dosovitskiy et al. 2020). while tabular variables are tokenized as in Gorishniy et al. (2022). This fine- fine-graine grained tok- enization enables experiments that isolate the contribution of specific inputs, such as specific tabular variables (e.g., BIO1: the annual mean temperature), groups of tabular variables (e.g., WorldClim), individual years of the climatic time series (c.g., 2018), multi-year segments (e.g., 2000-2018) 8), and specific patches (e.g., the center patch) of the satellite image.
AUC at each epoch, with
A small constant such as 0.001 can he added 8m: dm, and fám. to prevent division by zero.

---

Published as a conference paper at ICLR 2026
Table I: AUC performance on the GeoPlant test set. Each column corresponds different input subset, showing the performance of each masking strategy on that subset. For each masking strategy, the same trained model is evaluated on all subsets. The best score per subset is written in bold, and the average score across input subsets is reported in the last column.
Input Type
Modality
Partial Unimodal
Unimodal
Bimodal
All
BIOI
Tabular
WorldClim
Others
Clim: 2018
V
<<<
Avg.
Time series
Clim: 2000-2018
Landsat
Center patch Others
Sat. image
Constant
68.6
82.4
84.7
86.7
55.1
83.3
90.0
63.6
90.0
83.3
89.2
87.9
80.4
Uniform
73.3
85.7
86.3
87.2
61.2
86.9
91.1
65.6
91.6
86.2
91.8
92.0
83.2
Dirichlet
65.1
82.7
77.8
86.8
54.9
87.5
91.1
58.2
91.8
88.6
91.7
91.4
80.6
Modality dropout
48.7
80.8
77.4
86.4
66.2
88.6
91.4
73.2
92.0
89.2
91.7
92.0
81.5
OPM
68.0
81.9
80.7
85.3
68.1
88.4
90.2
81.1
90.7
89.5
91.1
91.2
83.8
MIAM (ours)
78.4
86.7
86.0
87.0
70.8
89.0
91.4
80.1
91.7
89.5
91.5
91.7
86.1
Oracle (one model per column) 78.0 87.1 87.7 87.6
77.1
89.3
92.2
81.4
92.3
89.7
91.7
92.0
87.2
TaxaBench (Sastry al., 2025) consists of species observations sourced from iNaturalist (Van Horn et al. 2018), each associated with five modalities: ground-level image, a satellite image, an audio recording, environmental tabular predictors, and the geolocation. The task is classify the species represented each sample. Since the dataset was originally designed for zero-shot classification, no predefined splits available. We therefore retain only species with least 10 observations (199 species) and stratify the remaining 4876 samples into train (80%), validation (10%), and test (10%) sets, ensuring the same proportions of species all splits. The validation loss at each epoch is used to compute per-modality scores, and we set X = and = 10 for MIAM. We reuse the pre-trained encoders from Sastry al. (2025). where each encoder outputs single token per modality. The resulting five tokens are then masked according to the masking strategy and passed to the transformer for the final prediction. Since we consider only one token per modality, the constant and modality dropout masking strategies are equivalent.
In Section 4.2.1
we compare MIAM against existing masking strategies and an oracle baseline: non-masking model trained directly on the exact subset of tokens under evaluation. Since the number of possible subsets grows exponentially, O(2Em= Tm), oracle models are impractical at scale. We therefore include them only as an approximate upper bound on performance for specific input subsets and under the given model and training setup. When restricted to a single modality, the oracle reduces to unimodal baseline. All models and baselines are trained under the same protocol for a given dataset (detailed in Appendix [. and A..), differing only their masking strategies. For each masking strategy, the same trained model is evaluated across all input subsets, meaning that no retraining is performed when switching subsets. An ablation and sensitivity analysis of MIAM are presented in Section 4.2.2 and Appendix A.4.I
4.2 RESULTS
4.2.
COMPARISON AMONG MASKING STRATEGIES
Table 1 shows the AUC performance of the different masking strategies on the GeoPlant dataset. Overall, MIAM outperforms all baselines, with an average gain of 2.3% over the second-best method. MIAM performs strongly across all unimodal setups, substantially narrowing the gap caused by modality imbalance, which is most evident for satellite imagery. OPM performs slightly better satellite imagery and some of the subsets it favors during training (e.g., satellite & tabular or satellite & time series). However, fails considerably on subsets never encounters, particularly partial unimodal setups, where MIAM remains robust and uniform masking is the main

---

Published as a conference paper at ICLR 2026
Top-1 accuracy the test of TaxaBench. Each column corresponds different input subset, showing the performance of each masking strategy on that subset. For each masking strategy, the same trained model is evaluated on all subsets. The best score per subset is highlighted in bold, and the average performance for each masking strategy across input subsets is reported in last column.
Table 2:
Input Type
Modality
Unimodal
Bimodal
Trimodal
Quadri. All
Ground-level image Audio
<rc
Avg.
Geographic location
<
<<
Environmental features
Satellite image
Uniform
42.4
41.2
8.40
7.99
6.76
59.2
48.8
64.3
9.02
51.2
46,9
65.8
37.7
Dirichlet
42.2
40.8
5.33
5.12
7.58
59.2
48.4
65.0
9.63
51.4
45.9
67.8
37.4
Modality dropout
41.4
39.8
5.53
4.51
8.2
57.2
44.3
59.2
9.63
51.0
45.1
65.0
35.9
OPM
33.2
35.0
5.74
5.12
7.79
46.3
34.4
50.0
10,9
43.6
42.6
59.4
31.2
MIAM (ours)
42.2
41.8
6.56
7.38
9.84
60.9
50.2
65.4
10.2
52.0
49.0
69.1
38.7
Oracle (one model per column)
45.3
44.9
7.58
9.43
12.9
63.3
50.0
66.6
13.1
51.8
46.5
69.
40.0
Time series only
Tabular only
Evolution Ps, with MIAM
Validstion AUL (16)
1.1
90
90
Method
1.b
85
85
MIAM
80
Beta hypercube
0.9
Uniform hypercube
75
75
Uniform
0.8
Satellite image only
All modalities
Evolution of Ps, with MIAM
Valication AUC (%)
95
Modality
80
90
Time series
15
Tabular
75
Satellite image
70
80
5
75
40
40
Epech
Epoch
Epoch
Figure 4: Ablation study and effeet of dynamic masking. Validation AUC on GeoPlant for each modality and intermediate masking strategies leading to MIAM (left), and evolution of MIAM's modality imbalance coefficients Ps... and pa... during training (right).
competitor. Yet, uniform masking does not account for modality dominance, resulting in poor per- formance on satellite imagery. When using all modalities, MIAM performs slightly below modality dropout and uniform masking. However, this gap is eliminated by reducing the strength of the imbalance-aware coefficients via X, which controls the tradeoff between dominant and dominated modalities (see Appendix A.r our hyperparameter analysis). Finally, the gap between MIAM and the oracle baseline is small, indicating that MIAM approaches the performance of specialized models trained directly each input subset, while remaining broadly flexible.
Table reports the t accuracy of different masking strategies on the TaxaBench dataset. On average, MIAM achieves the strongest performance, particularly in multimodal settings. The only exception is the trimodal case excluding the dominant ground-level image and audio, where OPM performs better since this subset appears most frequently during its training. In unimodal cases, both MIAM and uniform masking remain strong basclines. Additional metrics (top-5 accuracy and Fl-score) reported in Appendix A.4.2 and show the same trends as the top-1 accuracy.
4.2.2 ABLATION STUDY
In the left side Fig. 4 we show validation performance on GeoPlant during training for uniform masking and the successive variants of MIAM: starting with the uniform hypercube (full-support
8

---

Published as a conference paper at ICLR 2026
80
NIR+Red
87.0
Test AUC (%)
Others
200 ur oppapaann Resturave
Test AUC (%)
86.8
70
86.6
86.4
86.2
NIR
Blue
86.0
Green
1
4
7
10
13
16
19
Red
Temporal context (years)
Performance across different spectral band combinations of satellite imagery
(b) Performance with increasing historical climatic context
Figure 5: Contribution analysis for ecological insights. MIAM enables fine-gre fine-grained contribution within modalities. (a) The Red and NIR bands are particularly important as they are used to compute vegetation indices such as NDVI. (b) Extending the temporal context (longer time series) captures signals from past extreme events, such as heatwaves.
principle), extending to the beta hypercube coomer-priorttization principle). and finally to MIAM (imbalance-awar principle). While performance is similar for the dominant time series modality and for all modalities combined, each added principle consistently improves results on the other modalities, most notably for satellite imagery. MIAM also displays cyclic patterns, driven by fluc- tuations the relative learning speed pa... which periodically shift the training focus across modal- ities. This link is evident in the correspondence between the Pa curve (right side of Fig. 4 and the validation performance curve of MIAM. We hypothesize that such cyclic effects are beneficial for learning, similar to periodic learning-rate -aae schedules. By contrast, the relative performance scores Ps... remain fairly stable throughout training: while they provide a useful prior for identifying dom- inant modalities, relying solely them, in OPM, leads to suboptimal performance because the masking distribution evolves little over time (see Appendix A.4.1 for numerical results).
Table illustrates the positive effect of using non- uniform corner weights w. which prioritize the cor- ners O..., 0) and (1, 1, 1................... 1). The table reports the average performance gain across the input sub- sets of Table [I and Table 2 on their validation sets. While the improvement is modest on GeoPlant, the gain on TaxaBench more pronounced potentially because it includes more modalities, making these key corners occur even less frequently under uni- form corner weights. Additional ablation studies and a sensitivity analysis of the hyperparameters are pro- vided in Appendix(A.4. A.4.I
Table 3: Average performance impact of the prioritization of key corners by using a non- uniform W.,, evaluated on the validation sets of GeoPlant (AUC) and TaxaBench (Top-1 accuracy).
GeoPlant
TaxaBench
Uniform we
85.2
36.0
Non- Non-uniform Wec
85.4
37.1
DISCUSSION
The strong performance of MIAM on incomplete multimodal data, both across and within modal- ities, enables more accurate estimation of input contributions. By comparing performance across modalities or subsets of tokens, we can identify which inputs drive performance and may play critical ecological roles. For example, in the GeoPlant dataset, we find that model performance consistently improves when the Red and NIR spectral bands of satellite imagery are used together (Fig, (a). This aligns with well-established knowledge in remote sensing and vegetation monitor- ing: Red and NIR bands are used to compute the Normalized Difference Vegetation Index (NDVI). which captures biomass and phenological patterns of vegetation (Pettorelli et al. 2005 He et al. results indicate that model performance generally improves with longer time series (Fig. (b). We observe a significant increase performance when the time series includes the 2003 European heatwave, highlighting the importance of including temporal scales that capture key
2015). In addition.

---

Published as a conference paper at ICLR 2026
information such as extreme events (Lynch et al., 2014 Fonteyn et al. 2025 However, analyses such as in Fig. rely on manual inspection of selected subsets, which becomes impractical as the number of tokens grows. To obtain systematic and interpretable measures, as future work, we plan to compute token-level Shapley values, yielding a single contribution score per token, thus extending prior approaches to multimodal settings (Covert et al. 2023 Zhinden al. 20266)
Additionally, the formulation of masking for multimodal learning developed in this work can serve as foundation for designing new strategies tailored to specific multimodal learning challenges. For instance, if particular token subsets are of higher importance than the others, the corner weights we can be adjusted accordingly. MIAM can also be beneficial for large multimodal models such as 4M (Mizrahi et al. (2023). which currently rely on the Dirichlet distribution and may suffer from modality imbalance during training. In particular, its advantages are most evident in datasets with many modalities and more than one token per modality. Our experiments on the bimodal SatBird dataset (Teng et al. 2023) show that, in such a simple setting, all masking strategies perform sim- ilarly (see Appendix A.4.3). This highlights the need for richer multimodal benchmarks, both in ecology and in machine learning more broadly, that incorporate more than two modalities.
Finally, while we focus supervised setups in this work, masking strategies are often used for self-supervised pre-training. However, to the best of our knowledge, modality imbalance has not been explored in the context of SSL. To assess the potential of MIAM in SSL setting, we conduct a small-scale, proof-of-concept experiment. Although the Beta hypercube variant (MIAM with x 0) can be implemented directly, MIAM requires per-modality performance scores, which are difficult to obtain in SSL because labels are absent. For this experiment, we estimate modality performance using the reconstruction losses computed on the training set. While alternative approaches could be explored, this choice is sim- ple and broadly applicable. The experimental design follows the MultiMAE framework (Bachmann et al., 2022), adapted to the modalities in GeoPlant. 'The model is pre-trained on the training set using a given masking strategy, linear weights for linear prob- ing (LP) are learned on the validation set, and evaluation across input subsets is performed on the test set. Full experimental de- tails and per-subset results are provided in Appendix [A.4.4] with summary shown in Table 4 Overall, MIAM outperforms the other masking strategies under LP, demonstrating its promise for self-supervised settings, with the uniform strategy (Zbinden al., 2026) performing comparably. Although this is a small-scale ex- periment, highlights the importance of exploring masking strate- gies in self-supervised multimodal learning and suggests that the commonly used Dirichlet distribution may not be optimal, given MIAM's consistent improvements.
Table 4: Average AUC per- formance of masking strate- gies on GeoPlant with SSL pre-training evaluated via lin- ear probing. Per-subset re- sults are provided in Table I0]
Method
Avg.
Constant
78.3
Uniform
79.3
Dirichlet
77.0
Modality dropout
77.5
OPM
75.3
MIAM (ours)
79.5
6 CONCLUSION
Leveraging multimodal learning with masking strategies essential in ecological applications, where data is heterogeneous, incomplete, and requires interpretable input contributions. By formal- izing masking strategies for multimodal settings, we introduce MIAM, a dynamic masking strategy that effectively covers the space of input subsets while addressing modality imbalance, central challenge in multimodal learning. Our results demonstrate that MIAM consistently outperforms ex- isting masking approaches across multiple ecological datasets, enabling models to handle arbitrary input subsets and yielding valuable ecological insights through fine- -ee-grained contribution analysis.
REPRODUCIRILITY STATEMENT
We include all necessary details to ensure reproducibility. Model architectures and experimental setups are presented in the main text and the appendix. The code and trained model weights are pub- licly available at attttps: //gttuutbbbb com/ http://orrobn///aa.... https: htth///hggaaaccee///e//////////e
co /bi robi MMMA
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the Swiss National Science Foundation under grant 200021 204057 "Learning unbiased habitat suitability at scale with Al (deepHSMM," and by the Horizon Europe grant 101213369 (DVPS). We thank Julie Charlet for her early contributions the time series com- ponent and Emanuele Dalsasso for his precious feedback.
REFERENCES
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi Multi-modal multi- task masked autoencoders. In European Conference on Computer Vision (ECCV), pp. 348-367. Springer, 2022.
Roman Bachmann, Oguzhan Fatih Kar, David Mizrahi, Ali Garjani, Mingfei Gao, David Griffiths, Jiaming Hu, Afshin Dehghan, and Amir Zamir. 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2): 423-443, 2018.
lan Connick Covert, Chanwoo Kim, and Su-In Lee. Learning to estimate Shapley values with vision transformers. In International Conference on Learning Representations (ICLR), 2023.
Aaron Defazio, Xingyu Yang, Ahmed Khaled, Konstantin Mishchenko, Harsh Mehta, and Ashok Cutkosky, The road less scheduled. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pp. 9974-10007, 2024.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American chapter of the associ- ation for computational linguistics: human language technologies, pp. 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2020.
Chenzhuang Du, Tingle Li, Yichen Liu, Zixin Wen, Tianyu Hua, Yue Wang, and Hang Zhao. Im- proving multi-modal learning with uni-moda teachers. arXiv preprint arXiv:2106. 1059, 2021.
Benoit Dufumier, Javiera Castillo Navarro, Devis Tuia, and Jean-Philippe Thiran. What to align in multimodal contrastive learning? In International Conference on Learning Representations (ICLR), 2025.
Jane Elith and John R Leathwick. Species distribution models: ecological explanation and prediction across space and time. Annual Review of Ecology. Evolution and Systematics, 40(1):677-697, 2009.
Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo. PMR: Prototypical modal rebalance for multimodal learning. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp.
2023.
Ward Fonteyn, Josep M Serra-Diaz, Bart Muys, and Koenraad Van Meerbeek. Incorporating cli- matic extremes using the GEV distribution improves SDM range edge performance. Journal of Biogeography, 52(3):780-79) 2025.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learn- ing models for tabular data. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 18932-18943, 2021.
Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tab- ular deep learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 24991-25004, 2022.
11

---

Published as a conference paper at ICLR 2026
Florian Hartig, Nerea Abrego, Alex Bush, Jonathan M Chase, Gurutzeta Guillera-Arroita Mathew A Leibold, Otso Ovaskainen, Loic Pellissier, Maximilian Pichler, Giovanni Poggiato, et al. Novel community data in ecology-propente and prospects. Trends in Ecology & Evolution, 39(3): 280-293, 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, 2022.
Kate S He, Bethany A Bradley, Anna Cord, Duccio Rocchini, Mao-Ning Tuanmu, Sebastian Schmidtlein, Woody Turner, Martin Wegmann, and Nathalie Pettorelli. Will remote sensing shape the next generation of species distribution models? Remote Sensing in Ecology and Conservation, 1(1):4 18, 2015.
Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, and Longbo Huang. Modality competition: What makes joint training multi-modal network fail in deep learning? (provably). Interna- fional Conference on Machine Learning (ICML), pp. 9226-9259. PMLR, 2022.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances Neural Information Processing Systems (NeurIPS), volume 25, 2012.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2017.
Heather J Lynch, Mare Rhainds, Justin M Calabrese, Stephen Cantrell, Chris Cosner, and William Fagan. How climate extremes not means defnne species' geographic range boundary via demographic tipping point. Ecological Monographs, 84(1):131-144 2014.
Zhongqi Miao, Yuanhan Zhang, Zalan Fabian, Andres Hernandez Celis, Sara Beery, Chunyuan Li, Ziwei Liu, Amrita Gupta, Md Nasir, Wanhua Li, Jason Holmberg, Meredith Palmer, Kaitlyn Gaynor, Pablo Arbelaez, Pengce Wang, Rahul Dodhia, and Juan Lavista Ferres. New frontiers in artificial intelligence for biodiversity research and conservation with multimodal language mod- els. Methods in Ecology and Evolution, 2025.
David Mizrahi, Roman Bachmann, Oguzhan Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4M: Massively multimodal masked modeling. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023.
Natalia Neverova, Christian Wolf, Graham Taylor, and Florian Nebout. Moddrop: adaptive multi- modal gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38 (8):1692-1706, 2015.
Xiaokang Peng. Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8238 888--227,222 2022.
Nathalie Pettorelli, Jon Olav Vik, Atle Mysterud, Jean-Michel Gaillard, Compton J Tucker, and Nils Chr Stenseth. Using the satellite-derived NDVI to assess ecological responses to environ- mental change. Trends in ecology & evolution, 200))503-510, 2005.
Lukas Picek, Christophe Botella, Maximilien Servajcan, César Leblanc, Rémi Palard, Théo Larcher, Benjamin Deneu, Diego Marcos, Pierre Bonnet, and Alexis Joly. Geoplant: Spatial plant species prediction dataset. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pp. 126653-126676, 2024.
Laura Pollock, Sara Beery, Kaitlyn Gaynor, Marta Jarzyna, Oisin Mac Aodha, Bernd Meyer, David Rolnick, Graham Taylor, Devis Tuia, Tanya Berger- Wolf, and Justin Kitzes. Harnessing AI to fill global biodiversity shortfalls. Nature Reviews Biodiversity, 1:166 16666882 2025.
David R Roberts, Volker Bahn, Simone Ciuti, Mark S Boyce, Jane Elith, Gurutzeta Guillera-Arroia, Severin Hauenstein, José J Lahoz-Monfort, Boris Schröder, Wilfried Thuiller, et al. Cross- validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecog- raphy, 40(8)9913-92, 2017.
12

---

Published as a conference paper at ICLR 2026
Srikumar Sastry, Subash Khanal, Aayush Dhakal, Adeel Ahmad, and Nathan Jacobs. Taxabind: A unified embedding space for ecological applications. JEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1765-1774,202.,
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, lya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1995) 2014.
Gencer Sumbul, Chang Xu, Emanuele Dalsasso, and Devis Tuia. SMARTIES: Spectrum- multi-sensor auto-encoder for remote sensing images. In IEEE/CVF International Conference Computer Vision (ICCV), pp. 5569-5578, 2025.
Mélisande Teng, Amna Elmustafa, Benjamin Akera, Yoshua Bengio, Hager Radi, Hugo Larochelle, and David Rolnick. Satbird: dataset for bird species distribution modeling using remote sens- ing and citizen science data. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023.
Shagun Uppal, Sarthak Bhagat, Devamanyu Hazarika, Navonil Majumder, Soujanya Poria, Roger Zimmermann, and Amir Zadeh. Multimodal research in vision and language: A review of current and emerging trends. Information Fusion, 77:149-171, 2022.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8769-8778, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NeurIPS), volume 30, 2017.
Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification net- works hard? In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12705, 2020.
Yake Wei, Di Hu, Henghui Du, and Ji-Rong Wen. On-the-fly modulation for balanced multimodal learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Nan Wu, Stanislaw Jastrzebski. Kyunghyun Cho, and Krzysztof J Geras. Characterizing and over- coming the greedy nature of learning in multi-modal deep neural networks. In International Conference on Machine Learning (ICML), pp. 24043-24055. PMIR, 2022.
Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey.
IEEE Transactions Pattern Analysis and Machine Intelligence, 45(10))12113-122
2023.
Robin Zbinden, Nina Van Tiel, Benjamin Kellenberger, Lloyd Hughes, and Devis Tuia. On the lection and effectiveness of pseudo-absences for species distribution modeling with deep learning. Ecological Informatics, 81:102623, 2024.
Robin Zbinden, Nina Van Tiel, Gencer Sumbul, Chiara Vanalli, Benjamin Kellenberger, and Devis Tuia. MaskSDM with Shapley values to improve flexibility, robustness and explainability in species distribution modelling. Methods Ecology and Evolution, 17(1):18-2006, 2026.
Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xi- angyu Yue. Meta-transformer: A unified framework for multimodal learning. arXiv preprint arXiv:2307. 2023.
Yongshuo Zong, Oisin Mac Aodha, and Timothy M. Hospedales. Self-supervised multimodal learn- ing: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 5297) 5318, 2025.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
The appendix contains the following information:
Datasets and training details (A.1)
MIAM marginal distributions and the rationale behind corner prioritization (A.
Baselines details A.3
Additional experiments (A.4): ablation study and hyperparameter analysis, extra metrics
TaxaBench, results SatBird, and an evaluation of MIAM in self-supervised learning pre-training context
LLM usage A.5
A.1 DATASETS AND TRAINING DETAILS
We provide additional information on the datasets and their associated training procedures. All baselines share the same training protocol and model architecture; the only difference lies in the masking strategy. The following details common both GeoPlant and TaxaBench. Models are trained with AdamW (Loshchilov & Hutter 2017) using a weight decay of 0.01, a schedule- free approach (Defazio al. 2024), learning rate of 0.001, batch of 128, and dropout rate of 0.1 (Srivastava et al. 2014). Training runs for 100 epochs with early stopping based on the average validation AUC across the unimodal setups and the multimodal setup with all modalities included.
The models consist of tokenizers that produce tokens for each modality, followed by a trans- former (Vaswani et al. (017) that fuses them. Tokens are masked according to the selected strategy, with masked tokens replaced by a learned mask embedding, and the resulting sequence is processed by a 3-block transformer with 8 heads, similar to (Gorishniy et al. 202T). The transformer outputs the same number of tokens, which averaged to obtain a final representation and then passed through a linear layer to produce the logits. The next two sections provide details specific to each dataset.
A.1.1 GEOPLANT
The geographic distribution of the data splits is shown in Fig. 6 and the code to reproduce the exact splits provided Fig. using the verde verdee for block cross-validation (Roberts al.,
2017).
Training
Validation Test
Figure 6: GeoPlant geographic distribution of the data splits. Blocks of size 1 1
We tokenize each input part (patch, time segment, or tabular variable) independently. This yields 76 tokens for the climate time scries (4 channels 19 years). 126 tokens for the Landsat time series (6 channels x 21 years), 100 tokens for Landsat patches (4 channels 25 patches), and 48 tokens for the 48 tabular variables. Patches and time segments are encoded following Dosovitskiy et al.) (2000) using linear projection match the token dimension along with sinusoidal positional embeddings.
https: www.. fat wttp:/aaaaaannb.og/eeerreeee
14

---

Published as a conference paper at ICLR 2026
import pandas pd import verde as vd
df
ant/
coordinates d[[["lat", *Ion"]] ttooouuuyyyy specing: flost 1. test_ size: float 0.15
val_size: float 0.25
train_block, test_black vd.traiin_eet__spltt coordinates traattssttaassssee Lodeex.oo_uuppry spacingrepacing, beet_sizenteet_s.22 randon_state-12,
train_indioes, test_ rest_'ndicas train_bllcc(( test_block([]
train_block, vai_block vd.trrinntest__ppl.
cocrdinates staiitiiiettinnnnceceeeeee [rsrinnaaddess)...oopasttt
train_indices,
spacingrapacing,
test_sizo-val__et
(1 test_aize),
randon_state=42,
indices indices black(] (0), val_block(1]
Figure 7: Python code for splitting the GeoPlant data using the verde library for block cross- validation (Roberts et al. 2017).
import pandas pd
from sklearn. model selection import train_test__spl from sklearn, preprocessing import LabelEacoder
df weedd_/cr(//aat//t//ttaaaeee
df [""scientific_nnn "].isin
pd.nataFrams www.vvlvo__eooooooooeeeeeeoleeee countttotttaaaaoaoeeoaoe toutttttttaatttaaoteeee -quuery"count) (*count se 10*)
Lindex
y LabelEncoder () www_traassfrrmiceee det.ee/mmmmdsscceiiiiisss.eee
x df.drop (columng= "scientific_namp"
X_train, X_remaLddes, y_train, y_remainder ccain
X, Yr test_size=0. stratisyry, sandon_esate-22,
test,, X_val, y_test, y_val prrain_tes=_ppp atll remmindess, y_remeinder, test_aizeed.,,, frrartff-=y_emmattee random_state=42,
Figure Python code for stratified splitting of species observations of TaxaBench.
Tabular variables are tokenized as in Gorishniy et al. (2021) by projecting each scalar into higher- dimensional space with periodic activation functions. All tokens have size 192. A final sigmoid is applied to the logits to predict the presence of 1783 species. The model is trained as multi-label classifier using the weighted binary cross-entropy loss from Zbinden al. (2024).
A.1.2 TAXABENCH
The code to generate the stratified splits is provided in Fig. We reuse the pre-trained encoders released by the dataset authors (Sastry al. 2025). Each encoder outputs single token of size 512 per modality, resulting in a total of five tokens. softmax function is applied to the final logits, and the model is trained with the cross-entropy loss.
15

---

Published as a conference paper at ICLR 2026
1.0
10-
0.5
6.0
0.0
0.5
0.5
1.0
(a) Bcta hypercube
(b) MIAM
Figure 9: Marginal distributions of the beta hypercube and MIAM. Same setup as in Fig. with three modalities where modality 3 is dominated, but here showing the marginal distributions of the beta hypercube and MIAM. Each panel shows the kernel density estimate 5000 sampled points. (a) Beta hypercube: mixture of product heta distribution to prioritize corners (b) MIAM (dynamic masking): beta hypercube with imbalance-awar adjustments, causing modality 3 to be masked less frequently than the others.
A.2 ADDITIONAL. MIAM INTUITIONS
A.2.1 MIAM MARGINAL DISTRIBUTIONS
We provide an additional visualization in Fig. 9 showing the marginal distributions of the beta hy- percube and MIAM, which offers further intuition into how MIAM operates.
A.2.2 CORNER PRIORITIZATION
The motivation for prioritizing the corners and for overweighting certain corners more than others stems from both ecological considerations and optimization behavior:
Prioritizing corners: In ecological applications, incomplete data often occurs the level of entire modalities (e.g., a satellite image at a given timestamp). Prioritizing corners re- flects this situation by emphasizing full-presence or full-absence cases. Moreover, tokens from the same modality tend to be highly correlated (e.g., spectral bands, temporal seg- ments), meaning that observing only few tokens may already capture most of the modal- ity's 's information. This can exacerbate modality imbalance if multiple tokens of the dom- inant modalities are always present, further motivating the need emphasize complete absence cases.
All-tokens corner: It also common in ecological datasets have (nearly) complete observations across all modalities. Ensuring strong performance in these frequent, fully observed cases crucial for overall accuracy and generalization. Prioritizing this corner also encourages the model to learn joint cross-modal interactions efficiently when all inputs are available, leveraging complementary information across modalities.
Few-tokens corner: Given the high intra-modality correlation, it is important that the model remains capable extracting useful information even when only a few tokens are available. This is particularly relevant for interpretability: if we wish to assess the contri- bution of a single token (c.g., a specific tabular variable), the model must have encountered situations in which the corresponding token appeared nearly alone during training.
16

---

Published as a conference paper at ICLR 2026
Table Ablation of MIAM components. Validation AUC on GeoPlant when different elements of MIAM are removed. Each column corresponds to different input subset, showing the performance of each masking strategy on that subset. For each masking strategy, the same trained model is evaluated on all subsets. The best score per subset is shown in bold and the average performance across all input subsets is reported the final column.
Input Type
Modality
Partial Unimodal
Unimodal
Bimodal
All
BIO1
Tabular
WorldClim
Others
Clim: 2018
Avg.
Time series
Clim: 2000-2018
Landsat
Center patch
Sat. image
Others
MIAM
73.1
85.3
85,5
86.0
70.9
88.4
80.1
88.9
91.7
91.9
91.7
85.4
MIAM w/o pren
67.7
85.4
85.0
86.2
70.1
88.0
91.5
80.1
88.8
91.4
91.7
91.5
84.8
MIAM w/o pdm
71.3
85.4
85.2
86.
70.1
87.8
91.8
76.4
88.7
91.
92.0
92.0
84.9
MIAM uniform we
73.0
85.2
85.3
85.9
70.6
88.3
91.4
79.5
88.9
91.3
91.6
91.4
85.2
While these motivations partly hypothesis-driven. observe empirical support for them. Fig. highlights the performance impact on dominated modalities (high difference between Beta and Uni- form hypercubes), and the ablation study in ssoos consistent performance gains especially for dominated modalities like satellite imagery (see Table 5) when using non-uniform corners weights w. Importantly, as mentioned in the discussion, MIAM remains flexible: for different ap- plications, we can be adapted to emphasize alternative corners that better reflect application-specific data patterns.
A.3 BASELINES DETAILS
We provide additional details on the baselines and explain why OPM constitutes a suboptimal mask- ing strategy.
Constant: the probability of masking each token is fixed at 0.75, following He et al. 2022
at every iteration, shared masking probability ~ u((0, 1) is sampled and applied across modalities.
Uniform:
Dirichlet: following Mizrahi al. 2023). we use a symmetric Dirichlet with a 1
Modality dropout: following Neverova et al. 20155). tokens from a given modality are masked with probability 0.1.
OPM: follow the definition from Wei et al. 2024 and adopt their hyperparameters,
Tbase 0.5 and À = 0.4. practice, this formulation produces largely fixed masking behavior during training, per-modality performance scores remain nearly constant. Re- lying solely on these scores can also be misleading, since low now-performing modalities may simply be uninformative rather than worth emphasizing. Another limitation is that domi- nated modalities are never masked, which prevents the model from being exposed to the full range of modality combinations. Finally, OPM, originally defined, only drops entire modalities, without enabling fine- grained masking within them.
A.4 ADDITIONAL EXPERIMENTS
A.4.1 ABLATION AND SENSITIVITY ANALYSIS
Table highlights the importance of incorporating both modality imbalance coefficients, Ps... and Pd..: removing either reduces performance by at least 0.5% on average. In particular, dropping Pa... substantially degrades the unimodal setup satellite imagery, underscoring its role in identifying
17

---

Published as a conference paper at ICLR 2026
92.5
90
Validation AUC (%)
90.0
87.5
82.5
75
80.0
70
77.5
65
10
20
A
Time series
Tabular
Clim: 2018
Center patch
Sat. image
All
B101
Figure 10: Sensitivity of MIAM to X and the GeoPlant dataset. The reported metric is the validation AUC as X and & are varied.
60
60
Top-1 accuracy (96)
50
50-
40
30
20
20
10-
10
Ground-level image
Geographic location
Satellite image
Audio
Environmental features
All
Figure 11: Sensitivity of MIAM to À and on the TaxaBench dataset. The reported metric is the validation Top-1 accuracy as À and k are varied.
dominated modalities. The same figure also shows positive effect from using non-uniform cor- ner weights Was which prioritize the corners (0, O) and II,I,..., and yield a modest but consistent improvement across all subsets.
MIAM depends on the hyperparameters and k. In Fig. 10, we observe that controls tradeoff between dominant and dominated modalities GeoPlant, with larger values improving the perfor- mance of dominated modalities, i.c., satellite imagery. The same figure demonstrates that к also variation
plays important role, particularly for setups with fewer tokens, where see performance depending on its value. However, a similar experiment on the TaxaBench dataset (Fig. [T] indicates that these hyperparameters are less critical in some settings (notably TaxaBench, which contains only one token per modality), as performance remains stable across different val- ues. We recommend setting K 10 and choosing À between I and depending on the degree of modality imbalance. Note that because À and к parameterize the masking distribution, their effects can be inspected directly by varying them and visualizing the resulting distributions (e.g.. Fig, and Fig. 19. without requiring model retraining. Naturally, they can also be fine-tuned like any other hyperparameter using a validation set.
In Table 6 we evaluate the masking strategies using higher-capacity models to assess whether our findings hold at larger scales. We consider a relatively large transformer variant (6 layers, 256- dimensional tokens) and relatively huge variant (12 layers, 512-dimensional tokens), and evaluate the masking strategies on them. Across both larger models, MIAM remains the strongest masking strategy average, outperforming the other baselines by clear margin. However, also observe a decline in overall performance as model size increases, which may indicate overfitting and suggest that additional data would be needed to fully leverage these larger models.
18

---

Published as a conference paper at ICLR 2026
Table 6: AUC performance the GeoPlant test set with increased model size. Model size is varied by adjusting the number of layers and the token dimension; the base configuration is the one used throughout the main paper (see T. Each column corresponds to a different input subset, showing the performance of each masking strategy that subset. For each masking strategy, the same trained model evaluated on all subsets. The best score per subset shown in bold, the best score per subset within each model size setting is underlined, and the average performance across all input subsets is reported in the final column.
Input Type
Modality
Partial Unimodal
Unimodal
Bimodal
All
BIO1
Tabular
WorldClim
Others
Clim: 2018
AJJ
Avg.
Time series
Clim: 2000-2018
J
Landsat
Center patch
Sat. image
Others
Base: layers, dimensions
Constant
68.6
82.4
84.7
86.7
55.1
83.3
90.0
63.6
90.0
83.3
89.2
87.9
80.4
Uniform
73.3
85.7
86.3
87.2
61.2
86.9
91.1
65.6
91.6
86.2
91.8
92.0
83.2
Dirichlet
65.1
82.7
77.8
86.8
54.9
87.5
91.1
58.2
91.8
88.6
91.7
91.4
80.6
Modality dropout
48.7
80.8
77.4
86.4
66.2
88.6
91.4
73.2
92.0
89.2
91.7
92.0
81.5
OPM
68.0
81.9
85.3
88.4
81.1
90.7
89.5
91.1
91.2
83.8
MIAM (ours)
78.4
86.7
86.0
87.0
70.8
89.0
91.4
80.1
91.7
89.5
91.5
91.7
86.1
Large: 6 layers, 256 dimensions
Constant
58.0
82.2
83.9
86.8
55.1
81.4
89.6
55.4
89.9
81.2
90.5
90.6
78.7
Uniform
68.7
84.5
86.9
65.3
90.4
66.6
90.7
91.4
91.6
82.9
Dirichlet
69.1
84.3
79.2
85.6
53.5
87.6
91.2
59.3
92.0
88.5
92.0
92.0
81.2
Modality dropout
66.0
81.9
74.5
86.1
65.0
88.4
91.5
73.7
91.9
88.9
91.8
91.9
82.6
OPM
62.0
82.4
68.1
86.1
69.5
86.8
88.6
80.0
88.8
89.0
90.1
90.2
81.8
MIAM (ours)
74.5
86.0
85.0
87.3
70.3
90.7 88.5
78.7
90.9
89.2
90.9
90.9
85.2
Huge: 12 layers, 512 dimensions
Constant
58.6
72.0
85.5
57.8
75.2
88.1
61.1
88.3
88.8
88.4
77.2
Uniform
69.6
77.4
84.2
85.6
62.4
81.8
86.5
64.3
87.3
83.7
88.9
89.2
80.1
Dirichlet
57.4
79.1
81.5
86.3
60.9
83.8
89.9
62.1
90.6
85.5
91.0
91.2
79.9
Modality dropout
54.0
77.8
72.8
86.4
65.0
87.0
88.7
70.5
89.8
88.0
89.7
90.3
80.0
OPM
62.1
81.0
73.7
81.5
66.0
84.4
82.9
75.4
84.9
87.1
86.9
87.5
79.4
MIAM (ours)
63.8
82.7
83.0
85.9
67.6
84.9
86.8
75.4
86.9
87.4
88.6
88.8
81.8
A.4.2
ADDITIONAL METRICS FOR TAXABENCH
We provide additional metrics for TaxaBench top-5 accuracy (Table 7) and F1-Score (Table complementing the top-1 accuracy shown in the main text. Interestingly, across all three metrics, masking strategies outperform the oracle certain modality subsets. We hypothesize that this occurs because the tokens, obtained from the pre-trained encoders of Sastry et al. (2025), are already well aligned, allowing masking strategies to exploit cross-modal features by exposing the model to more tokens. By contrast, the oracle is limited to tokens from the evaluated subset and cannot benefit from this cross-modal information. addition, the relatively small dataset makes results more variable, and masking may provide regularizing effect that improves generalization.
A.4.3 SATBIRD
We explore the SatBird dataset (Teng al. 2023 benchmark for species distribution models with only two modalities: tabular environmental predictors and Sentinel-2 satellite imagery. With just two modalities, the performance differences between baselines minimal, making the dataset less useful for evaluating masking strategies. Nevertheless, we include it here for completeness and transparency.
19

---

Published as a conference paper at ICLR 2026
Table 7: Top-5 accuracy on the test of TaxaBench. Each column corresponds different input subset, showing the performance of each masking strategy that subset. For each masking strategy, the same trained model evaluated on all subsets. The best score per subset is written in bold, and the average score across input subsets is reported in the last column.
Input Type
Modality
Unimodal
Bimodal
Trimodal
Quadri.
All
Ground-level image Audio
<
Avg.
<<<
Geographic location
<<
Environmental features
Satellite image
Uniform
65.6
67.4
21.5
22.5
84.0
71.7
87.3
72.7
88.3
58.7
Dirichlet
65.6
64.5
19.1
17.2
25.2
84.2
72.5
85.9
26.0
75.8
75.2
91.0
58.5
Modality dropout
63.1
62.3
15.8
18.4
21.5
80.1
68.4
83.2
23.0
74.4
71.9
86.9
55.8
OPM
60.0
59.8
13.3
17.2
24.2
62.1
77.3
28.9
71.3
69.5
87.1
53.8
MIAM (ours)
66.0
65.0
24.0
22.7
25.0
83.4
73.8
25.2 88.3
75.8
72.1
88.3
59.1
Oracle (one model per column) 64.1 67.0
21.3
28.3
31.1
84.2
72.3
87.1
31.8
74.2
75.8
89.5
60.6
Table 8: F1-score on the test set of TaxaBench. Each column corresponds to a different input subset, showing the performance of each masking strategy on that subset. For each masking strategy, the same trained model is evaluated on all subsets. The best score per subset is written in bold, and the average score across input subsets is reported the last column.
Input Type
Modality
U Unimodal
Bimodal
Trimodal Quadri. All
Ground-level image
Audio
<<<
Avg.
Geographic location
v
<<<
Environmental features
Satellite image
Uniform
35.9
31.7
6.74
3.26
6.05
50.1
42.7
57.4
6.16
45.5
38.8
59.4
32.0
Dirichlet
34.1
29.1
3.17
2.86
5.79
43.3
41.7
52.4
5.94
46.4
38.4
61.0
30.3
Modality dropout
34.1
27.5
3.81
2.87
5.50
45.6
37.2
49.3
5.32
44.3
37.4
58.4
29.3
OPM
21.6
20.9
1.51
3.11
5.43
27.4
23.9
32.6
8.03
35.9
30.2
47.7
21.5
MIAM (ours)
33.7
31.9
4.13
3.92
49.1
45.1
55.4
6.93
46.9
42.0
61.2
32.2
Oracle (one model per column) 38.9
5.85
4.56
5.05
50.7
43.0
56.2
6.80
44.9
36.9
60.3
32.3
Table 9: SatBird dataset results across metries. Each column corresponds different input subset, showing the performance of each masking strategy on that subset. For each masking strategy, the same trained model is evaluated on all subsets. The best score for each subset is written in bold.
Modality
MAE [x 10c2] 4.
Top-10 acc.
Top-30 acc.
Environmental variables
Avg.
Avg.
Avg.
Satellite image
Dirichlet Modality dropout
1.93
2.26
2.13
2.11
47.2
28.5
35.9
37.2
62.9
56.0
60.0
59.6
2.02
2.26
2.05
2.11
45.4
34.5
42,9
40.9
61.4
56.7
60.8
59.6
OPM
1.97
2.21
2.11
2.09
46.3
33.5
39.1
39.7
62.2
56.2
58.9
59.1
MIAM (ours)
1.94
2.24
2.12
2.10
45.6
32.6
40.8
39.7
62.0
57.2
61.1
60.1
Oracle (one model per column)
1.91
2.20
2.03
2.05
47.8
30.6
42.0
40.1
63.2
54.8
63.4
60.5
20

---

Published as a conference paper at ICLR 2026
Table 10: AUC performance of masking strategies on the GeoPlant test set with SSL pre- training under linear probing. Each column corresponds to a different input subset, showing the performance of each masking strategy on that subset. For each masking strategy, the same trained model is evaluated on all subsets. The best score per subset is written in bold, and the average score across input subsets is reported the last column.
Input Type
Modality
Partial Unimodal
Unimodal
Bimodal
All
BIO1
Tabular
WorldClim
Others
Clim: 2018
Avg.
Time series
Clim: 2000-2018
Landsat
Center patch Others
Sat. image
Constant
70.9
80.5
81.6
82.3
65.0
82.8
84.5
68.1
84.6
76.9
80.6
81.5
78.3
Uniform
70.9
81.8
82.5
82.7
66.4
83.4
84.9
69.2
85.2
79.9
81.8
82.4
79.3
Dirichlet
68.5
82.0
78.6
82.4
59.7
83.6
83.3
64.7
83.5
80.1
78.4
79.6
77.0
Modality dropout
70.9
79.2
81.1
81.9
63.9
82.5
84.8
68.7
83.9
81.2
77.4
74.9
77.5
OPM
70.8
78.1
79.2
81.0
58.8
79.3
81.2
62.8
83.1
77.1
74.9
76.7
75.3
MIAM (ours)
70.8
81.7
81.7
81.3
65.7
83.9
85.5
70.6
85.8
82.7
82.0
82.1
79.5
Unlike GeoPlant, SatBird focuses on 670 bird species and formulates the task as a multi-target regression problem, where the model predicts encounter rates (i.c.. values between and 1) each location. We use the same Summer train/validan vrinn/vaiiatioon/tsst splits as in Teng et al. (2023). Satellite features are extracted using the same ResNet-18 pretrained on ImageNet (Krizhevsky et 2012) as in the original work. For the tabular variables, we use 3-layer MLP with hidden dimension 256 and ReLU activations. Both networks produce 512-dimensional vectors, which are concatenated and passed through linear layer followed by sigmoid to yield the final predictions. The mask token set to zero vector. The training procedure follows the code provided by [Teng et (2023). For MIAM and OPM, modality scores computed the inverse of the validation loss.
The results in Table report Mean Absolute Error (MAE) along with top-10 and top-30 accuracy. Overall, masking strategy consistently outperforms the others: all achieve relatively low MAE, and the best method varies across metrics and modality combinations without a clear trend. As noted earlier, with only two modalities and single token per modality, masking strategies provide little added value, making this setting less informative for evaluation.
A.4.4 MIAM FOR SUPERVUPE TRE-TNGINN
MIAM designed for supervised tasks, but its applicability may extend beyond this setting. this section, evaluate the potential of MIAM in self-supervised pre-training context. The pretext task consists of reconstructing the values of each modality in the GeoPlant dataset, i.e., the tabular variables, time-series entries, and Sentinel-2 image pixels. The transformer encoder and tokenizers follow the same structure as in the main supervised GeoPlant experiment (detailed in Appendix A. After encoding, the tokens are passed -ppelic decoders reconstruct each modality. Following MultiMAE (Bachmann et al. 2022). but adapted to non-image modalities, we use one decoder per modality. Each decoder consists of layer normalization. cross-attention layer, another layer normalization, a feedforward network, three transformer blocks, final linear projection, and a reshape operation to match the format of the target modality. Learned positional embeddings are added to the tokens before entering the decoders allow the decoder to capture both token position and token type.
This multimodal autoencoder is then trained to reconstruct the modality values using mean squared error loss, averaged first within each modality and then across modalities. Pre-training is performed on the GeoPlant training using the given masking strategy that determines which tokens to hide, following the same procedure as in the main supervised experiment (details are in Appendix I. We then evaluate the quality of the learned embeddings using linear probing (L.P.. We follow the
21

---

Published as a conference paper at ICLR 2026
procedure of SMARTIES (Sumbul et al. 2025 : linear classifier is trained for 100 epochs with a learning rate 0.001 and batch size 1024. The linear weights are learned on the GeoPlant validation set using weighted binary cross-entropy loss optimized with Adam, and performance is then evaluated across the different input subsets.
Since MIAM requires computing per-modality score update its modality coefficients, we use the reconstruction losses during training as a proxy for modality performance. This choice may not be optimal, but straightffrward to implement across different setups. We leave the exploration of more principled alternatives to future work.
reported in Table TO Two masking strategies stand out: the uniform strat- egy (Zbinden et al., 2026) and MIAM. MIAM performs best when all modalities are present, i.c., in unimodal and bimodal settings, whereas the uniform strategy performs slightly better in partial unimodal setups. Notably, traditional masking strategies for SSL,, such as the constant and Dirichlet strategies, underperform, suggesting that they may not well suited for learning robust representa- tions under missing inputs. These findings indicate that stronger masking strategies tailored to SSL are needed, and that MIAM provides promising direction.
The results
A.5
LLM USAGE
We used Large Language Models (LLMs) assistive tools during the preparation of this manuscript. Their role was limited to polishing grammar, improving clarity, and providing feedback on the co- herence of our ideas. They were not involved in generating research questions designing experi- ments.
22


Reviews:
[{"review_id": "mFFHs3sb6e", "reviewer": "Reviewer_Cimh", "summary": "This paper proposes a new masking algorithm that addresses the problem of modality competition, wherein one modality dominates the learned features. The algorithm uses a mixture of product beta distributions that concentrations probability mass in the corners (especially the all-on or all-off corners) of a unit hypercube representing the probability that tokens within each modality are masked. The weights are adjusted dynamically to adjust the masking probability depending on the performance and learning speed of each modality. Experiments are performed on two multimodal ecology datasets, GeoPlant and TaxaBench, using a transformer architecture. MIAM overall improves performance on these benchmarks compared to baselines including On-the-fly Prediction Modulation (OPM), which adjusts per-modality probabilities based on relative performance scores but applies the probability to the entire modality, not each token within each modality as in MIAM. The paper also shows how MIAM indicates which inputs drive performance, providing a measure of the contribution/importance of each modality.", "questions": "- Why is it beneficial to prioritize combinations with almost all tokens or almost no tokens from each modality?\n- How does MIAM affect the performance of multimodal remote sensing foundation models?\n- How does applying the same probability to all tokens in a modality compare to applying a different probability to each token regardless of modality?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- This paper addresses an understudied problem in multimodal learning of modality imbalance/competition. Despite being understudied, it is a common challenge in multimodal learning for remote sensing (and other domains).\n- The paper is well written and easy to read. Technical details are clearly explained. Figures and tables are high quality and are helpful for understanding the paper’s results and ideas.\n- The contribution analysis enabled by MIAM is a nice bonus and would be appreciated by domain experts in ecology and other domains (e.g. across remote sensing applications).\n- The ablation experiment is well designed and effectively shows the contribution of each component of the proposed algorithm.\n- The proposed algorithm is well-motivated, and the paper gives nice explanations and figures to support the motivation and intuition behind the algorithm’s design.", "weaknesses": "- The MIAM masking algorithm could be applied to any model architecture that implements masked multimodal learning. I think there was a missed opportunity to show the value of MIAM on existing multimodal remote sensing foundation models. If it worked, MIAM could significantly improve the utility of these models. (To me, this is the difference between a score of 8 and 10.)\n- It seems that a natural extension (or even baseline?) of MIAM is to assign a masking probability to each token, rather than applying the same probability to all tokens within each modality. Did the authors consider or test this?\n- I think the motivation for prioritizing the all-on/all-off corners (or corners in general) could be better explained in terms of the ecological application context. It doesn’t seem obvious to me why it would be beneficial to prioritize combinations with almost all tokens or almost no tokens from each modality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H6AX1ok2u0", "reviewer": "Reviewer_5e4q", "summary": "This paper introduces MIAM (Modality Imbalance-Aware Masking), a dynamic masking strategy for multimodal learning that addresses the challenge of modality imbalance in ecological applications. The key insight is to formalize masking strategies as probability distributions over unit hypercubes and design a principled approach with three properties: (i) full support over all input combinations, (ii) corner prioritization to favor complete/minimal modality combinations, and (iii) imbalance-awareness that adaptively masks dominant modalities based on their performance and learning dynamics. MIAM uses a mixture of product beta distributions whose parameters are dynamically adjusted during training based on per-modality performance scores and their temporal derivatives. The method is evaluated on two ecological benchmarks (GeoPlant and TaxaBench) and demonstrates consistent improvements over existing masking strategies while providing fine-grained contribution analysis.", "questions": "See above", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Testing on two diverse ecological datasets (GeoPlant with 3 modalities, TaxaBench with 5 modalities) with multiple modality combinations demonstrates robustness.\n2. The fine-grained contribution analysis (Fig. 5) demonstrates how the method can provide ecological insights (e.g., importance of NDVI bands, impact of extreme events), bridging ML and domain science.\n3. The progressive ablation showing the contribution of each design principle (uniform hypercube → beta hypercube → MIAM) is convincing.", "weaknesses": "1. Only ecological datasets are thoroughly evaluated; broader applicability claims need support from other domains. The SatBird result (Appendix A.4.3) showing similar performance across strategies raises questions about when MIAM is beneficial. No comparison on standard multimodal benchmarks (e.g., vision-language tasks)\n2. The choice of ε=3 and φ=10 appears arbitrary. THere is limited discussion of how to set these in practice. The corner weights (Eq. 3) use specific fractions (1/4, 1/2) without justification\n3. OPM (Wei et al. 2024) is the main dynamic masking baseline, but other recent modality balancing methods are mentioned but not compared. Missing comparison with recent self-supervised multimodal methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CmyOfPw3tI", "reviewer": "Reviewer_CJxa", "summary": "The paper introduces MIAM (Modality Imbalance-Aware Masking), a dynamic masking strategy for multimodal learning. MIAM formalizes masking as probability distributions over unit hypercubes and addresses three key principles often missing in prior methods: Full support for all input combinations, Corner prioritization to emphasize critical configurations, and Imbalance awareness by adapting masking probabilities based on modality dominance. To achieve this, MIAM constructs a mixture of product beta distributions and dynamically adjusts masking during training using modality-specific performance and learning speed. This design enables handling arbitrary missing inputs, mitigating modality imbalance, and supporting fine-grained contribution analysis across and within modalities.", "questions": "-\tCould MIAM be adapted for other model architectures? If so, what could be challenges? \n-\tHow should practitioners choose λ and κ in practice for new datasets without extensive tuning?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-\tInteresting approach to improve performance and robustness when modalities are missing. \n-\tStrong average performance across different subsets of modalities compared to other sampling strategies. \n-\tMethodology is clearly described and well-structured. \n-\tProvides insightful analysis of how modality-specific values evolve during training.", "weaknesses": "-\tOnly tested on two classification downstream tasks. Broader applicability (e.g., segmentation tasks like flood mapping where optical data is often missing) is not demonstrated. Also includes frequent references to pre-training masking strategies in related work, while not compared for pre-training.\n-\tDownstream performance is sensitive to new hyperparameters. With ablation shown only on one dataset (Figure 10), it is difficult for readers to tune parameters effectively. \n-\tEvaluated with only one architecture, which is not well explained, even though MIAM is likely applicable to many others. \n-\tTraining details and architecture description are insufficient. Before presenting results, it should be made explicit that one model per masking strategy was trained and then evaluated on different subsets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JBVz2ry2by", "reviewer": "Reviewer_Hfi3", "summary": "The paper introduces a masking strategy called MIAM. The objective is to improve the robustness of transformers to missing modalities at test time. The topic is important because there is a trend towards fusing more and more modalities, while it is crucial to maintain performance on any subset of modalities seen during training for practical utility. MIAM increases the probability of a modality being masked (not given as input) if its unimodal validation performance is high and not changing; conversely, modalities with low and increasing accuracy are masked less often (given as input). MIAM improves over other masking strategies for two multi-modal datasets.", "questions": "None", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Multi-modal masked modelling is increasingly used, so the topic is important\n- I learned from the paper and it may inform my research\n- The idea of MIAM is intuitive, fairly simple, and seems to work — at least when the number of modalities > 2\n- The paper is well written and has nice figures that aid understanding", "weaknesses": "Moderate weakness:\n- I believe that according to the appendix, all transformers have 3 layers and embedding dimension 192, which is very very small. For reference, a ViT-Base has 12 layers and dimension 768. I understand that larger models are costlier but I suspect model size will interact with masking strategies. For example, larger models may be able to fit all modalities easier, and thus uniform masking may be fine. \n\nMinor weakness:\n- The method seems to rely on computing validation accuracy after each epoch. Thus it cannot be directly used in self-supervised learning, e.g., masked autoencoding (MAE), since we aim to learn task-agnostic representations without using labelled data. Since multi-modal MAE is quite popular, having MIAM directly support MAE would be nice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)