Test-Time Adaptation for Combating Missing Modalities in Egocentric Videos
Authors: Merey Ramazanova
Alejandro Pardo
Bernard Ghanem
Motasem Alfarra
Venue: ICLR 2025 Poster (2025)
Keywords: missing modality
test-time adaptation
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose a Test-Time Adaptation (TTA) method specifically tailored for the problem of missing modalities in multimodal egocentric recognition. Our method outperforms other TTA baselines when modalities are missing at test time.

Abstract: Understanding videos that contain multiple modalities is crucial, especially in egocentric videos, where combining various sensory inputs significantly improves tasks like action recognition and moment localization. However, real-world applications often face challenges with incomplete modalities due to privacy concerns, efficiency needs, or hardware issues. Current methods, while effective, often necessitate retraining the model entirely to handle missing modalities, making them computationally intensive, particularly with large training datasets. In this study, we propose a novel approach to address this issue at test time without requiring retraining. We frame the problem as a test-time adaptation task, where the model adjusts to the available unlabeled data at test time. Our method, MiDl~(Mutual information with self-Distillation), encourages the model to be insensitive to the specific modality source present during testing by minimizing the mutual information between the prediction and the available modality. Additionally, we incorporate self-distillation to maintain the model's original performance when both modalities are available. MiDl represents the first self-supervised, online solution for handling missing modalities exclusively at test time. Through experiments with various pretrained models and datasets, MiDl demonstrates substantial performance improvement without the need for retraining.

Published as a conference paper at ICLR 2025
TEST-TIME ADAPTATION FOR COMBATING MISSING MODALITIES IN EGOCENTRIC VIDEOS
Merey Ramazanova, Alejandro Pardo, Bernard Ghanem, Motasem Alfarra Center of Excellence in Generative AI, KAUST, Saudi Arabia fi rstnamm. w nawe@aaust.eeeemmmm
ABSTRACT
Understanding videos that contain multiple modalities is crucial, especially in egocentric videos, where combining various sensory inputs significantly improves tasks like action recognition and moment localization. However, real-world appli- cations often face challenges with incomplete modalities due to privacy concerns, efficiency needs, or hardware issues. Current methods, while effective, often ne- cessitate retraining the model entirely to handle missing modalities, making them computationally intensive, particularly with large training datasets. In this study, we propose a novel approach to address this issue at test time without requiring retrain- ing. We frame the problem to the available unlabeled data at test time. Our method, MiDI (Mutual information with self- elff-Dstilllation), encourages the model to be insensitive the specific modal- ity source present during testing by minimizing the mutual information between the prediction and the available modality. Additionally, we incorporate self-distillation to maintain the model's 's original performance when both modalities are available. MiDI represents the first self -upervieeddd online solution for handling missing modalities exclusively at test time. Through experiments with various pretrained models and datasets, MiDI demonstrates substantial performance improvement without the need for retraining. Our code is available in this repo.
test-time adaptation task, where the model adjusts
INTRODUCTION
Understanding multimodal data has emerged as pivotal challenge in various domains, including foundational model construction (Radford et al., 2021), emotion recognition (Lee et al., 2019), and analysis of egocentric videos (Grauman et al., 2023; 2022; Damen et al., 2018 8) for tasks like recog- nition (Kazakos et al., 2021; Nagrani al., 2021; 1; Xiao et al., 2020) and localization (Ramazanova et al., 2023; Wang et al., 2023a). Recent efforts focused on crafting models harnessing data diverse modalities showing performance gains in multiple tasks (Lin et al., 2022). However, a critical limita- tion arises: these models presuppose complete modality availability at test-time, which diverges from real-world scenarios where data modalities may be incomplete (Ma et al., 2022; Ramazanova et al., 2024). For instance, in real-time prediction using wearable devices, portions of recordings might be redacted to safeguard privacy, or cost constraints may necessitate using cheaper modalities like audio or IMU (Grauman et al., 2022; 2023). Consequently, models designed under this assumption demonstrate significant performance degradation in the face of missing modalities, sometimes even underperforming unimodal counterparts trained with a single modality (Ramazanova al., 2024).
The challenge of addressing missing modalities gained significant attention from researchers recently, where various strategies have been proposed to mitigate this issue (Neverova et al., 2015; Ma et al., 2021; 2022; Ramazanova et al., 2024). Some works have explored architectural modifications to fuse information from different modalities effectively (Ma et al., 2022). Additionally, other approaches have focused on designing effective regularizers to boost model performance when confronted with missing modalities (Colombo et al., 2021). More recently, a promising direction has emerged, wherein transformer models have been augmented with learnable tokens during training (Lee et al., 2023; Ramazanova et al., 2024). These tokens serve to compensate for missing information at test time, significantly enhancing model robustness against missing modality (Ramazanova et al., 2024). Despite these advancements, a common drawback persists: all existing approaches necessitate expensive retraining of the multimodal model, rendering pretrained models obsolete. This poses

---

Published as a conference paper at ICLR 2025
SD
g(fo,)
S
fo
No Adaptation
Adaptation with MiDI
Missing Video dx Missing Audio
Figure I: Test-Time Adaptation for missing modalities. The concept of test-time adaptation in the presence of missing data modalities focuses on system where a stream of multimodal data is input, potentially lacking one or more modalities. Without adaptation, the pretrained model fe, may predict inaccurate labels due to incomplete data. With test-time adaptation, the model is dynamically adjusted using the adaptation method 9, resulting in an adapted model fe., designed handle the missing modalities and improve over time. The graph on the right illustrates the performance of the non-adapted baseline (blue) vs. the model adapted with our proposed adaptation method MiDI (green) on Epic-Kitchens dataset. It shows the adaptation efficacy in maintaining higher performance levels despite the variability in modal-completeness surpassing the unimodal performance (orange) for all missing rates.
substantial challenge, particularly in applications with extensive training data, where the retraining process is prohibitively expensive, making the aforementioned approaches impractical. Consequently, a fundamental question arises:
Can we develop methods to address missing modalities at test time without imposing retraining requirements?
In this work, we take the initial step of framing the missing modality problem as a test-time adaptation problem (Wang et al., 2020; Liang et al., 2020; Li et al., 2016). Specifically, we aim to establish a new approach wherein pretrained models undergo adaptation at test time to optimize their performance in the presence of missing modalities. Our formulation assumes a scenario where a continuous stream of unlabeled data is fed into the pretrained model during testing, with some instances missing certain modalities. The objective is to devise an adaptation algorithm capable of refining the model's predictions under missing modality in real-time settings (refer to Figure 1). Based this formulation, we first evaluate existing methodologies from the test-time adaptation literature and demonstrate their limited efficacy in addressing this specific multimodal challenge. Subsequently, we introduce a novel test-time adaptation technique explicitly tailored to tackle the missing modality problem. Our method revolves around incentivizing the output of the pretrained model to remain invariant to the modality source present during testing. To achieve this, we propose minimizing the mutual information between the model's output and the modality type of the incoming unlabeled data at test time in a self-supervised manner. Moreover, to ensure the preservation of model performance under a complete modality setup, we integrate our approach with a self-distillation mechanism. Notably, our method, termed MiD1 (Mutual information with self- olf--istillltton minimization), is theoretically motivated and agnostic to the choice of pretrained model architecture, the dataset, and the specific type of missing modality encountered during testing.
We summarize our contributions in two-fold: (1) We redefine the missing modality problem as a test-time adaptation challenge, pioncering a novel approach where pretrained models adapted at test-time to optimize performance in the face of missing modalities. We evaluate the effectiveness of the current adaptation methods under this challenging problem. (2) We introduce MiDI (Mutual information with self-Distillation minimization), a versatile test-time adaptation method explicitly designed address the missing modality problem. MiDI ensures that model outputs remain invariant to modality sources during testing, enhancing robustness. It is agnostic to factors such as the pretrained model architecture, training dataset, and the specific type of missing modality, making it a comprehensive solution for diverse scenarios. When combined with pretrained models, MiDI achieves significant performance improvements, including a 6% gain on the Epic-Sounds dataset and an 11 11% % gain on the SppicKKithhen dataset.
2 RELATED WORK
Missing Modalities in Multimodal Datasets. Several works addressed the problem of missing modality in the multimodal datasets (Tsai et al., 2018; Ma al., 2021; Zhao et al., 2021; Neverova
2

---

Published as a conference paper at ICLR 2025
et al., 2015; Ma et al., 2022). Most methods addressing the missing modality problem assume full access to the source (training) data. Some works are based on the assumption that the training data is modal-compllte, and the goal is to train model robust to the missing inputs at test time (Ma et al., 2021; 2022). For example, Dai et al. (2024) investigate a strategy of randomly dropping video frames during training to improve the robustness of multimodal system. Similarly, Lee et al. (2019) propose a method to train network capable of generating audio features to handle missing modalities. Wang al. (2023b) focus on a multimodal learning approach that models shared and specific features for classification and segmentation tasks. Other works tackle the modality distillation task, where the training data is multimodal, but only one modality is used test time (Radevski et al., 2023; Garcia et al., 2019). Few works assume the modalities could be missing at train and test times and attempt to train a robust network (Lee et al., 2023; Ramazanova et al., 2024). In our work, we explore a more realistic scenario where we might not have access to the training data or network re-training is not feasible. We formulate this setup as a test-time adaptation problem, where the model is experiencing a distribution shift caused by the unavailability of some modalities at test time. Further, propose the first test-time adaptation algorithm tailored to combat the missing modality challenge at test-time.
Test-Time Adaptation. Test-time Adaptation (TTA) attempts combat performance gaps that pretrained models suffer from when exposed to distribution shifts at test-time (Mancini et al., 2018; Kojima al., 2022). This is usually attained through modifying the model's parameters (Liang et al. 2020) or its input (Gao et al., 2022) by using the incoming unlabeled data at test-time. TTA methods are practical, as they avoid assumptions on the training phase of given model (Wang et al., 2020). The first of these approaches adjusts the statistics of the Batch Normalization (BN) layers (Li et al., 2016). This was followed by more powerful adaptation methods that involved self-supervised objective functions such as entropy minimization (Wang et al., 2020; Niu et al., 2022; Niu14 al., 2023), information maximization (Liang et al., 2020), teacher-student approaches (Yuan et al., 2023), and leveraging auxiliary tasks (Alfarra et al., 2025). While such TTA methods made significant progress towards combating distribution shifts at test-time, they solely focused on simple covariate shifts such changes in weather conditions or pixel illumination (Hendrycks & Dietterich, 2019). In this work, we extend the problem formulation of test-time adaptation to a very practical and realistic domain shift: missing modality. In particular, we adapt the stream setting of online test-time adaptation (Alfarra et al., 2023) to formulate the missing modality problem along with the corresponding evaluation protocol. Building on this novel view of the missing modality problem, we analyze the current tife--of-hh-aar TTA methods where we show their limited impact. Further, we propose a novel TTA method tailored to combat the missing modality problem.
MISSING MODALITY AS TEST-TIME ADAPTATION
We first formulate the missing modality problem test-time adaptation problem in Section 3.1. We then outline the evaluation protocol for a given adaptation method in Section 3.2.
3.1 PROBLEM FORMULATION
In this work, we focus on the recognition problem. Let So Rd + P()) be a classifier that maps a given input x E Rd into the probability simplex P(Y") over the set of labels y = {1, (1,.....,K) KJ' In this work, we assume that an input is a multimodal input. However, in many realistic applications. the input provided to the model might have missing modalities, thus containing either audio only. visual only, or audio-visual information. Let m E (A, V, AV) denote the type of available modality for given input æ corresponding to audio only, visual only, audio-visual, respectively. For simple formulation, we fix the dimensionality of the input x by replacing the missing modality part with zeros. Further, we assume that fa is pretrained multimodal model on training set D. In this work, we make no assumptions on fo (i.e. choice of architecture), the dataset D. nor the training process.
At test time, fe is presented with a stream of unlabeled data S with possibly missing modalities. The likelihood with which a certain modality appears in data revealed from S is characterized by probability mass function; denoted by (Ms m). For example, if P = V) = 0.5 and PsMM = AV)- = 0.5, then the audio missing rate in the test stream is 50%. In other words, half the
data arrives as video only, without its accompanying audio. Let Pm. Ps PM M = m), then the missing
"egg the output after softmax layer.
3

---

Published as a conference paper at ICLR 2025
rate of different modalities can be equivalently characterized with P (P. (p..Pv..2 Pv.).v). Thus, for a stream with 25% missing video. P = {0.25, 0.0, 0.75) (i.e. the stream will reveal data with 25% probability of having only audio and 75% probability of revealing both modalities). According to this characterization, one can define the missing rate of at least one modality as PAv with pA being the rate of missing video and pv being the audio missing rate. Next, discuss the online evaluation protocol of fe under the stream S of unlabeled data.
3.2 EVALUATION PROTOCOL
Given our formulation of missing modality in Section 3.1, we are now ready to outline the evaluation protocol. Note that an adaptation method is function g(0) that sequentially adapts the model's parameters 9 to enhance the performance under the missing modality setup. Formally and following the online learning literature (Cai et al., 2021; Ghunaim al., 2023; Alfarra et al., 2023), we simulate the interaction between the stream S characterized by P and the TTA method y. at each time step t E {0, .., ) as follows:
1. S reveals a sample/batch ra with its corresponding modality m.
2. fe, generates the prediction it-
3. adapts the model parameter O, to 0c+1-
where fe, is the non-adapted pretrained model. That is, for each revealed sample/batch It, the model needs to predict its label before receiving the next data point I++1- The adaptation method g can exploit the predicted label to improve the model performance performance of an adaptation method g is measured in an online manner by comparing the predicted label Úe with the ground truth label yt-
the next revealed samples. The
PROPOSED SOLUTION
This section proposes our novel adaptation strategy to combat missing modalities at test time. Recall that the adaptation method g has to satisfy the following requirements. Firstly, g has to be fully self-supervised. The test stream reveals strictly unlabeled data at test time. Secondly, g has to conduct the adaptation in an online manner. That is, 9 should adapt on each revealed sample/batch of data æ, since S reveals I only after the model predicts Dt. (refer to the Evaluation Protocol in Section 3.2).
To formulate adaptation method, we begin by asking the following question: How should an optimal fo behave under missing modality? We believe that a robust model against missing modality should satisfy two properties. First, the prediction of fa should be invariant the modality source m. Ideally, fo should output the same prediction under both complete and incomplete modality, hence satisfying the following equality: (i) fo M A) fo(a;; M V) fo M = AV) Vi. (ii) fo should retain high performance in predicting data with complete modality, which is generally satisfied for Seg Satisfying both properties will result in model that accurate (satisfying (ii) and robust against missing modality (satisfying
(i)). To construct an adaptation algorithm that satisfies both properties, we propose to solve the following optimization problem:
8" arg min
[MI (fo m). m) + KL (fo(x) M = fV (1 M AV)]
(I)
where MI(u, 2) is the mutual information between u and v and KL is the KL-Divergence. Note that the mutual information between two random variables MI(u, v) 0, then u and are independent of each other. That is, minimizing the first term in the objective function in equation aims to satisfy property (i). Hence, if MI (Jo- (z; m), m) = O, then the output of the adapted classifier becomes independent from the available modality at test time. Furthermore, to ensure that the adapted parameters are still performing well upon adaptation, we equip the mutual information minimization with seff-distillation approach through minimizing KL divergence between the prediction of the adapted model and the original fe,, satisfying property (ii).
Although the objective function in equation 1 is self-supervised, obtaining 0* requires accessing all samples from the stream to evaluate the expectation Ez-s, which is not available at test time in the online evaluation protocol. To that end, approximate the expected value during
4

---

Published as a conference paper at ICLR 2025
Adapted model at step Adaptation with MiDI
5
fo.
LKL
feo
fe.
L
MMI
fe.
S
O.t1 = 0. -Vo--V + CKL)
Figure 2: Adapting at test-time with MIDI. At test time, the stream reveals a sample. MiDI uses multimodal samples to adapt and requires one forward pass for each modality combination. MiDI leverages (KL) divergence to align the predictions of the adapted model fo, with those of the original model fe., ensuring that the adapted model does not deviate too far from the original model's s predictions. The Informainooo (MI) component uses the prediction from the different modalities to reduce the dependency on any specific modality, fostering a more generalized and robust prediction across different modality combinations. MiDl updates the model for step + 1 using the combination of KL and MI in Equation
adaptation at time with the samples x, revealed from the stream. Hence, our Mutual information with self-Distillation (MiDI) adaptation step timestamp & can be expressed as the following:
oc+1- 0: Vo LANDI = 02 (o(M + Lou.)loo
with LKL KL (fo(ze; M = A AV)
K fo(m)) log (fa(1z))
Saln; ())mm)) log (fo( m)) LMI Em Σ
Σ
Lor
Lan
where fo(x.)) = Emm[fo(s m)], fa m) is the 7th element in the vector folses m), and 0 is the learning rate the gradient descent step.
To estimate the expectation Em [fe( m)], we conduct three forward passes for He with setting m E {A, AVA... We average these predictions to calculate Laiv and we average their entropy to calculate fem (refer to Figure 2). Note that Lai is the entropy of the average prediction across modalities. Note that under incomplete modality, Lemt = Lav resulting in LMi = 0. Hence, Lxi * 0 only when 7; has complete modalities (refer to Appendix A for details). Based on that, propose to conduct our adaptation step only when S reveals 2, with complete modalities. However, when S reveals v, with incomplete modality, we refrain from adaptation and 0,+1 O.. This makes the interaction between the stream S and our proposed MiDI at each time step take the following form:
1. S reveals a sample/batch ia with its corresponding modality
2. fe, generates the prediction Út-
3. If æ, is with complete modalities then adapts the model parameter 0g to 0.+1 through equation 2, else set 0,,1 8..
For 2t with missing modality, we leverage the most recent adapted model to perform predictions without adaptation. Since this work focuses on the multimodal setting, we assume that PAv * This setup aligns with real-world scenarios where multimodal data streams are common, and some modal-ccmplett instances are expected. Note that adapting a multimodal model at test time to an unimodal stream is particularly challenging without labeled data. Nevertheless, to demonstrate that MiDI does not degrade the original multimodal model's performance this extreme case, we also report results with PAV 0.
Takeaway. We are the first to formulate the missing modality challenge as a test-time adaptation problem. Our work makes no assumptions about the training phase in terms of architecture training objectives. We only require a model that works with multiple modalities at test time. We proposed MiDI, the first test-time adaptation method that effectively combats the missing modality challenge. MiDI updates the model only when it encounters poodality-complle samples but generates predic- tions for all samples, regardless of the modalities they contain. MiDI operates on any pretrained
5

---

Published as a conference paper at ICLR 2025
multimodal model at test-time by adapting its parameters on the received unlabeled data in an unsupervised online manner.
5 EXPERIMENTS
We thoroughly analyze MiDI's performance under different missing modality scenarios. We present our experimental setup in Section 5. In Section 5.2, we compare MiDI on a simple TTA scenario where the model has adapt while replying to the stream. In Section 5.3, we study MiDI's behavior under the Long- Term Adaptation (LTA) setup that happens when the model has been exposed to long stream of data S. Finally, Section 5.4 analyzes the scenario, in which one has access some unlabeled data from an out-of-doman source before the model is deployed. Under all these three scenarios, we show that MiDI the better alternative to combating missing modalities test time.
5.1
SETUP
We follow Ramazanova et al. (2024) and use validation sets of Epic-Kitchens and Epic-Sounds and report action accuracy. We use Ego4D-AR videos for the out-of-domain experiments Section 5,4.
Datasets. Epic-Kitchens (Damen al., 2018; 2022) contains 100 video hours of people performing cooking activities recorded with wearable cameras. The dataset is commonly used for benchmarking audiovisual egocentric action recognition. Each instance is annotated as noun and verb pair (e.g.. cut tomato), with a total of 300 noun and 97 verb classes. The validation set contains 9668 samples. Epic-Sounds (Huh et al., 2023) provides sound-based annotations for the same 100 video hours. has 44 classes and 8045 validation samples. We stick to the official train/val/test splits provided for both datasets. The approximate ratios are 75% for training, 10% for validation, and 15% for testing.
We assess the effectiveness of the baselines and our proposed MiDI in combating missing modalities at test time. To do so, we present the pretrained model with a stream Syal of unlabeled validation data where resulting in a missing rate of {100%, 75%, 50%, 25%, 0%), respectively. Following Ramazanova et al. (2024), for each dataset, we drop the primary modality (i.e. sound for Epic- Sounds and video for Epic-Kitchens). Thus, for a 75% missing rate (i.e. PAv 0.25), the stream containing Epic-Sounds has P {0.0, 0.75, 0.25), and the for Epic-Kithenes has P (0.75, 0.0, 0.25). We also report the performance of unimodal models that rely solely on the available modality (e.g., video for Epic-Sounds and audio for Epic-Kine Beic-Kitchenn). Ideally, a well-ad multimodal model should match or exceed this performance.
drop
modality with rate of 1 PAV. We set PAv E {0.0, 0.25, 0.5, 0.75, 1.0}
Architecture. Unless stated otherwise, we use the stronger architecture Multimodal Bottleneck Transformer (MBT) (Nagrani et al., 2021) as fo. Yet, we experiment with the vanilla self-attention architecture (Vaswani et al., 2017) in Section 6.1. Each backbone is fine-tuned on the training set of the corresponding dataset.
5.2 MIDL IMPROVES PERFORMANCE
Table compares our proposed MiDI against three off-the-shelf TTA methods. Namely, we include the information maximization approach, Shot (Liang et al., 2020), and the state-of-the-art entropy minimization with data selection from ETA (Niu et al., 2022) (refer to Section B. for details).
We observe that: (i) MiDI significantly enhances the baseline performance under missing modality. We record significant gain of 5% and 7% on Spic--Kichene with missing rates of 50% and 75%, respectively. In Epic-Sounds, MiDI boosts the accuracy of the baseline from 37.1% 1% and 28.3% to 38.8% and 29.8% under the same missing rates, respectively. Note that this performance boost comes at no cost of retraining the model; simply adapting during test time. This result demonstrates how our proposed mutual information mininization encourages the model to become invariant to domain shifts due to missing modality. (ii) MiDI successfully retains the baseline performance when all modalities are present, with accuracies of 55.0% and 63.7% at 0% missing rate Epic-Sounds and Epic-Kitchens respectively. This demonstrates that our proposed KL divergence regularization effectively preserves the information retention capability of the baseline under modal-complete

---

Published as a conference paper at ICLR 2025
Table 1: Combating missing modalities at test time. The first rows show the unimodal performance and the MBT baseline with no adaptation. We show three alternative TTA methods and demonstrate that our proposed MiDI is effective at combating missing modalities test time, outperforming all presented TTA baselines. Refer to Table to see the standard deviations.
PAV
Epic-Sounds (%)
Epic- BipicKKicheens (%)
Model
0
25
50
75
100
0
25
50
75
100
Unimodal
41,4
41,4
41.4
41.4
41.4
40.0
40.0
40.0
40.0
40.0
Baseline
55.1
45.6
37.1
28.3
19.5
63.9
55.5
46.8
37.9
29.5
+Shot
55.0
45.6
37.1
28.5
20.0
63.9
55.9
47.9
40.6
34.3
+Tent
54.8
45.0
35.9
26.5
17.8
63.7
54.0
39.2
24.2
9.9
+ETA
55.1
45.6
37.1
28.3
19.5
63.5
51.3
33.7
20.6
7.9
+MiDI (ours)
55.0
46.8
38.8
29.8
19.5
63.7
58.4
52.4
46.4
29.5
Table 2: Adaptation at Test-time under Long-term Adaptation and with Ego4D warm-up.
LTA. We showcase the results of MiDI. under the assumption that the stream of data is very long. We use unlabeled data to simulate a longer stream and report results on the validation set of each dataset. Our MiDI benefits from long-term adaptation, especially higher missing rates (>75%). Ego4D warm-up. We show another use cause MiDL, in which the assumption is having access to out-of-domain unlabeled data to adapt before deployment. The results showcase MiDL's capabiliiiess on leveraging ooteeleed-uo-o----o-oe data to combat missing modalities.
PAV
Epic Sounds (%)
Epic Kitchens (%)
Model
0
25
50
75
100
0
25
50
75
100
Baseline
55.1
45.6
37.1
28.3
19.5
63.9
55.5
46.8
37.9
29.5
+MiDI
55.0
46.8
38.8
29.8
19.5
63.7
58.4
52.4
46.4
29.5
MiDI I.TA
54.9
46.8
39.5
32.6
26.0
63.7
58.4
52.4
46.7
41.4
Ego44 Warm-up
55.0
46.5
38.6
30.4
20.4
63.7
58.4
52.4
46.7
37.8
inference. (iii) We also observe that presented TTA methods are less effective. This limitation arises because TTA methods designed to tackle covariate domain shifts and, thus, are not tailored to enhance performance under this specific type of domain shift (missing modality).
5.3 PERFORMANCE UNDER LONG-TERM ADAPTATION (LTA)
Next, we analyze the effectiveness our proposed MiDI under a long stream of data S, Since MiDI operates test time, its performance gain can vary depending the amount of data revealed at test time. Note that for any Pav * 0, as t - oc, MiDI would be exposed to large amount of unlabeled data with complete modality; enhancing the invariance properties of the adapted model against missing modality.
To study this interesting setting, we present MiDI with Sm followed by Sval used in Section 5.2. In this scenario, we allow MiDI to access unlabeled data with complete modality from Sm. to only perform adaptation. Then, we assess the efficacy of MiDI by performing adaptation and evaluation on Syal. We then ask the following question: how would MiDI perform after this long adaptation? We let Sn be subset of training data, and test the model Sta being the validation set, following the standard evaluation in Section 5.2.
Table summarizes the results on Epic-Sounds and PitccKKichenns datasets at the same missing rates considered in Section 5.2. We observe: (iv) the longer the stream is, MiDI provides bigger performance gain. For example, MiDI further improves the non-adapted baseline by 4.3% and 8.8% on Epic-Sounds and addition, even under 100% missing rate, MiDI improves the accuracy by a notable margin of 6.5% and 11.9% Epic-Sounds and Epic-Kitchent respectively. That is, the adaptation on S,n unlocks bigger potential of MiDI for life-long adaptation even when Syal reveals data with a single modality. (v) Unlike MiDI, other adaptation approaches do not benefit from this long stream setup as their objective functions do not promote building an invariant model against missing modality. We present these results in Table 7.
respectively, under a missing rate of 75% (i.e. PAv 0.25). In
7

---

Published as a conference paper at ICLR 2025
5.4 WARM-UP ON EGO4D: EXPLOITING OUT OF DOMAIN ADAPTATION
Next, we analyze another practical setup in which the model can be warmed up with some available data before deployment. particular, we consider the case when not only the pretrained model so provided, but unlabeled data from out-of-domain denoted as Sout can also be accessible. Given this setup, we wonder: would warming up with MiDI on a different data distribution Sout help combat missing modalities in Seat?
To answer this question, we leverage the recent Ego4D (Grauman et al., 2022) data. Although Ego4D has egocentric videos, they come from very different environments and scenarios that deviate from the usual kitchen scene. These changes introduce additional domain shifts when evaluating on Epic-Sounds and Epic-Kitchena We set Somt to be 5,000 clips of the Ego4D-AR training set. It is worth noting that we keep using our self-supervised MiDI objective, and we do not require any labels from Ego4D-AR.. We our setup from Section 5.3 and perform adaptation
Sou followed by the standard evaluation (Section 5.2) on Syal. We refer to the adaptation on Sout as the warm-up phase.
Table 2 summarizes the results where we show the performance of the non-adapted baseline, our proposed MiDI when adapted solely on Syal, and our MiDL equipped with warm-up adaptation on Sout We observe that (vi) conducting warm-up generally positively influences overall performance in cases of missing modality. The enhanced version of MiDI with Ego4D warm- up improves over MiDD by 0.6% and 0.3% on Epie-Sounds and Kitchens respectively, under missing rate of 75%. Furthermore, we observe that even under 100% missing rate, adapting on Saut enhances the aceuracy of MiDI by an impressive 8% on Epic- Ditc-Kischens. This demonstrates the versatility of MiDI, which provides consistent performance gains under different setups.
Takeaway. In this section, showcased the effectiveness of our proposed MiDI in combating the missing modality challenge at test time (Section 5.2). Further, we analyzed the impact of long-term adaptation where MiDI provided further performance gain where the relative improvement is up to 30% (Section 5.3). At last, we showed how adapting with MiDI on out-of-distribution data, mimicking data scareity situations, still boosts the model's accuracy (Section 5.4).
6 ANALYSIS ON MIDL
In this section, conduct comprehensive analysis of MiDI. In particular, show how proposed test-time adaptation is agnostic to the choice of fo (Section 6.1), the missing modality (Section 6.2). We conclude by analyzing the different components of MiDI in Section 6.4 and its computational requirements Section 6.5.
6.1 AGNOSTIC ARCHITECTURE CHOICE
Here, we analyze the robustness of MiDI against the architecture choice fe- To this end, replicate experimental setup from Sections 5.2 and 5.3 but set fo be the multimodal fusion via vanilla self-attention architecture (Vaswani et al., 2017), as opposed to the MBT architec- ture (Nagrani et al., 2021).
Table 3: MiDL performance with self-attention base- line. We showcase the effectiveness of MiDL with multi- modal self-attention. MiDL enhances performance across all missing rates, underscoring its robustness and adapt- ability to various underlying architectures.
1 - PAV
SppicSSonns (%)
Model
25
50
75
100
Table 3 summarizes the results under dif- ferent missing rates. We compare the non-adapted baseline, Shot and ETA, and our proposed MiDI. We observe that MiDI provides a consistent performance boost under the self-attention architec- ture, similar to our observations with the MBT architecture in Section 5.2. For ample, MiDI improves the accuracy of the baseline under 50% missing rate by a notable 1.1% without affecting the performance under complete modality.
Self-Att Att. Baseline
45.3
38.8
32.7
26.7
20.5
+Shot
45.5
39.0
32.8
26.8
20.7
+ent
45.3
38.6
32.3
26.0
19.8
+ETA
45.3
38.8
32.7
26.7
20.5
+MiDI (ours)
45.5
39.5
33.8
27.5
20.5
+MiDI LTA (ours)
45.5
39.6
34.5
29.0
23.2
8

---

Published as a conference paper at ICLR 2025
Table 4: Adaptation at Test-time- Other Missing Modalities. In this table we show the results using the complementary modality for each of the datset, ie. video for Epic Sounds and Audio for Epic Kitchens. We observe that MiDL improves consistently under this setup, highlighting its robustness to different types of modalities missing at test time.
PAV
Epic Sounds (%)
Epic Kitchens (%)
Model
0
25
50
75
100
0
25
50
75
100
Unimodal 46.5
46.5
46.5
46.5
46.5
63.2
63.2
63.2
63.2
63.2
Bascline
55.1
53.4
51.8
50.5
48.8
63.9
61.0
58.0
54.8
52,1
+MiDI
55.0
53.3
51.8
50.7
48.8
63.7
61.6
59.1
55.9
52.1
+MiDI LTA
55.0
53.4
52.0
51.0
49.4
63.7
61.7
59.5
57.3
55.3
Addtionally, Table presents the results under Long-Term Adaptation (LTA), following the setup in
Section 5.3. Similar to earlier findings, LTA unlocks better potential of MiDI where further performance gain is attained. Under this setup. MiDI improves the baseline by 2% under 50% and 75% missing rates, and by 2.7% under 100% missing rates. These results show that MiDI is agnostic to the choice of architecture.
6.2 AGNOSTIC TO THE TYPE OF MISSING MODALITY
In previous sections, we focused our experiments on the effectiveness of MiDI when the dominant modality is missing. In partieular, we analyzed dropping the audio modality for Epic-Sounds and the visual modality for Epic-Kitchen: In this section, we attempt to study the robustness of MiDI against dropping the non-dominant modality test time. Thus, in these experiments, we drop the visual modality in Epic-Sounds and audio in Epic-Kitchens. In contrast to the scenarios with missing dominant modality, these baselines experience less performance degradation. We replicate our experimental setup from Section 5.3 where we compare the non-adapted baseline, its equipped version with our proposed MiDI, and the LTA effect MiDI.
Table 4 summarizes the results. Our experiments show that MiDI consistently enhances the perfor- mance the pretrained model irrespective of the type of missing modality. For example, MiDI improves the baseline by 1% Epic Kitchens under missing rates of 50% and 75%. Further, and similar to our observations in Sections 5.3 and 6. 1, the LTA further improves MiDI by over 3% under 100% missing rate (i.e. PAv 0). In Table 9, demonstrate how MiDI generalizes "mixed modality" setup, where either modality could be missing at test time.
6.3 AGNOSTIC TO PRETRAINING
In previous sections, we adopted the baseline from Ramazanova et al. (2024), which uses masked autoencoder pretraining. To demonstrate how MiDI generalize to the different pretraining mechanisms, we apply MiDI on the Omnivore backbone (Girdhar et al., 2022). Note that Omnivore is modality-agnostic vision model with leverages several visual modalities in one unified architecture during pretraining. During testing, Omnivore is used for single-modaile modality downstream tasks on any visual modality (Girdhar al., 2022). Thus, we use omnivore initialize each one of our backbones. We report the results in Table 5. We observe how MiDI provides significant performance boost over the Omnivore baseline, when presented with missing modalities at test time. For example, under the 25% missing rate, MiDI improves the baseline performance by 9.5%, from 48. 1% to 57.6%. These results demonstrate that MiDI maintains its robustness and adaptability across different pretraining strategies, significantly improving the Omnivore baseline. This, yet again, highlights MiDI's generaizzation capability.
6.4 COMPONENTS OF MIDL
At last, we ablate the effectiveness of each component of MiDI. Recall from our adaptation step in equation that MiDl has two components: mutual information minimization (Mi) and self- distillation through minimizing KL-divergence (DI). We believe that the success of MiDI attributed to the interplay between both components.

---

Published as a conference paper at ICLR 2025
Table MiDI performance with Omnivore pretraining. MiDI is highly effective when applied to Omnivore model, demonstrating its effectiveness with different pretraining strategy.
- PAv
Spic-Kitchens (%)
Model
2,5
50
75
100
Omnivore Baseline
65.6
48.1
47.6
46.0
44.2
+MiDI (ours)
65.6
57.6
52.4
47.5
44.2
Table 6: Analyzing MiDI components. We analyze the different components of MiDL. When the Nutual-Information (MI) component is missing. the model does not have any reason to adapt since the KL divergence is maximized by predicting the same the base model. When KL is not present, the MI alone deviates from the initial results and performs poorly under higher missing rates.
PAV
LMI
LXL
Epic Sounds (%)
Epic Kitchens (%
Model
25
50
75
25
50
75
Baseline
55.1
45.6
37.1
28.3
63.9
55.5
46.8
37.9
+ DI
X
/
55.2
45.6
37.1
28.3
63.9
55.5
46.8
37.9
+ Mi
V
x
40.4
39.3
36.1
29.6
53.5
50.5
47.6
45.9
+MiDI (ours)
55.0
46.8
38.8
29.8
63.7
58.4
52.4
46.4
To analyze the importance of each component, we adapt the baseline with each loss function independently and compare the performance. Table 6 summarizes the results. We observe that adapting solely with self-distillation (i.e. minimizing LKL) results in no adaptation. On the contrary, adaptation by minimizing only LMi can result in significant performance drop for low missing rates. While minimizing Lxi can indeed result in fo that is robust against missing modality; might be less accurate for modal-complete samples (i.e. satisfying the first (i) property in Section 4 and violating the second (ii)). Note that for high missing rates, LMI can result in performance boost as a small number of adaptation steps will enhance the invariance properties while not causing significant divergence from the original model. MiDI balances information minimization with the information retention loss LKL. providing consistent performance gains under all missing rates.
6.5 COMPUTATIONAL REQUIREMENT OF MIDL
While our results have demonstrated the efficacy of MiDl in providing performance gains, we note that this improvement comes at computational cost. fact, conducting an adaptation step with MiDI requires 3 inferences through fo, and an inference through the initial pretrained model fe,- followed by a single backward pass. This makes an inference through MiDI 5 x more expensive than doing inference without any adaptation. In practice, the latency of MiDI is only 2x slower than the non-adapted model since all the additional 4 forward passes can be performed in parallel.
Takeaway. this section, we conducted comprehensive analysis on our proposed MiDI. We showed that MiDI is agnostic to the choice of architecture (Section 6.1) and the type of missing modality (Section 6.2), and the type of pre-training (Section 6.3). We further analyzed the importance of both components of MiDI in Section 6.4 and its computational requirements in Section 6.5. MiDI shows remarkable performance across the board, consistently delivering strong results regardless of the dataset scenario.
CONCLUSIONS
In this work, we presented MiDI, a new method for improving how pretrained video recognition models handle missing modalities at test time. MiDI improves the model's 's ability to give accurate predictions regardless of the availability of modalities by minimizing mutual information and using self-distillation. Our experiments show that MiDI can significantly increase accuracy across various datasets and scenarios and under various missing rates, making a practical solution for real-world applications dealing with incomplete modalities.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
The research reported in this publication was supported by funding from KAUST Center of Excellence on GenAL, under award number 5940.
REFERENCES
Motasem Alfarra, Hani Itani, Alejandro Pardo, Shyma Alhuwaider, Merey Ramazanova, Juan C Pérez, Zhipeng Cai, Matthias Müller, and Bernard Ghanem. Revisiting test time adaptation under online evaluation. arXiv preprint vrXiv:2304.04700 2023. 3, 4
Motasem Alfarra, Alvaro Correia, Bernard Ghanem, and Christos Louizos. Test-time adaptation with source based auxiliary tasks. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL wwtphhtps//// wwtp//wenneeee.eeeeeeeeeeeeee.we net/ fontttforurnt forum? 1d-ixAXXxXXXe 4n.
Zhipeng Cai, Ozan Sener, and Vladlen Koltun. Online continual learning with natural distribution shifts: An empirical study with visual data. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8281-8290, 2021.
Pierre Colombo, Emile Chapuis, Matthieu Labeau, and Chloé Clavel. Improving multimodal fusion via mutual dependency maximisation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih ). Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 231-245, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v 2.118653//122221.co 1/2021.emmlppppppp. -emnlp-main.n.. main.21. URL https: http:///aaaaaaaahaaeae
org 222g222211122 emnlp- ma mam.n................. 21.
Yusheng Dai, Hang Chen, Jun Du, Ruoyu Wang, Shihao Chen, Haotian Wang, and Chin-Hui Lee. A study of dropout-induced modality bias on robustness to missing video frames for audio-vi audiolvisual speech recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27445-27455, 2024. 3
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In European Conference Computer Vision (ECCV), 2018. 1,6 6
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100, International Journal of Computer Vision, pp. 1-23, 2022. 6
Jin Gao, Jialing Zhang, Xihui Liu, Trevor Darrell, Evan Shelhamer, and Dequan Wang. Back to the source: Diffusion-drivee test-time adaptation. arXiv preprint arXiv:2207.00 00342, 2022. 3
Nuno C Garcia, Pietro Morerio, and Vittorio Murino. Learning with privileged information via
adversarial discriminative modality distillation. IEEE transactions on pattern analysis and machine
intelligence, 42(100)2581-255 2019. 3
Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Hammoud, Ameya Prabhu, Philip HS Torr, and Bernard Ghanem. Real-time evaluation in online continual learning: A new paradigm. arXiv preprint arXiv:2302.01047, 2023. 4
Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. 2022 ieee. In CVF Conference on Computer Vision and Pattern Recognition. New Orleans, Louisiana, (June (June 2022), https://arxiv.
orgfabs/2201.08777 2022.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022. 1,8

---

Published as a conference paper at ICLR 2025
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. arXiv preprint (.Xiv:2311.182) 2023. 1
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representations, 2019. 3
Jaesung Huh, Jacob Chalk, Evangelos Kazakos, Dima Damen, and Andrew Zisserman. Epic-sounds: A large-scale dataset of actions that sound. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023. 6
Evangelos Kazakos, Jaesung Huh, Arsha Nagrani, Andrew Zisserman, and Dima Damen. With a little help from my temporal context: Multimodal egocentric action recognition. In British Machine Vision Conference (BMVC), 2021.
Takeshi Kojima, Yutaka Matsuo, and Yusuke Iwasawa. Robustifying vision transformer with- out retraining from scratch by test-time class-conditional feature alignment. arXiv preprint 1.Xiv:2206.13951,, 2022. 3
Hu-Cheng Lee, Chih-Y Yu Lin, Pin-Chun Hsu, and Winston H Hsu. Audio feature generation for missing modality problem in video action recognition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3956-39000 3960. IEEE, 2019. 1,
Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen Yu Lee. Multimodal prompting with missing modalities for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14943-14952, 2023. 1,3
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint 1rXiv:1603.04770 2016. 2, 3
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesís transfer for unsupervised domain adaptation, In International Conference on Machine Learning, pp.
PMLR, 2020. 2, 3, 6, 14
Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric 7. XU, Difei Tu, Wenzhe Zhao, Weijie Kong, et Egocentric lideo-language pretraining. Advances in Neural Information Processing Systems, 35:7575-7586, 2022. 1
Gao,
Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2302-2310, 2021. 1,
Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust to missing modality? Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18177-18188 18186, 2022. 1,3
Massimiliano Mancini, Hakan Karaoguz, Elisa Ricci, Patric Jensfelt, and Barbara Caputo. Kitting in the wild through online domain adaptation. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1103-1100 1009. IEEE, 2018. 3
Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems, 34: 14213, 2021. 1, 6,
Natalia Neverova, Christian Wolf, Graham Taylor, and Florian Nebout. Moddrop: adaptive multi- modal gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38 (8):1622-1706, 2015. 1,2
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International conference on machine learning, pp. 16888-16905. PMLR, 2022. 3, 6,
12

---

Published as a conference paper at ICLR 2025
Shuaicheng Niul4, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan15. Towards stable test-time adaptation in dynamic wild world. International Conference on Learning Representations, 2023. 3, 14
Gorjan Radevski, Dusan Grujicic, Matthew Blaschko, Frani-er Moens, and Tinne Tuytelaars. Multimodal distillation for egocentric action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5213-5224, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp.
PMLR, 2021.
Merey Ramazanova, Victor Escorcia, Fabian Caba, Chen Zhao, and Bernard Ghanem. Owl (observe, watch, listen): Audiovisual temporal context for localizing actions in egocentric videos. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp.
2023. 1
Merey Ramazanova, Alejandro Pardo, Humam Alwassel, and Bermard Ghanem. Exploring missing modality in multimodal egocentric datasets. arXiv preprint arXiv:2401.11470, 2024. 1, 19
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical stafistics, pp. 400-0000 407, 1951. 14
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhut- dinov. Learning factorized multimodal representations. arXiv preprint arXiv:1806.068 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 6,
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020. 2, 14
Hanyuan Wang, Majid Mirmehdi, Dima Damen, and Toby Perrett. Centre stage: Centricity-basse audio-visual temporal action detection. In The 1st Workshop in Video Understanding and its Applications (VUA 2023), 2023a.
Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carneiro. Multi- modal learning with missing modality via shared-specific feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15878-15887, 2023b.
Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audiovi- sual slowfast networks for video recognition. arXiv preprint arXiv:2001.08744 2020.
Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2023. 3
Jinming Zhao, Ruichen Li, and Qin Jin. Missing modality imagination network for emotion recog- nition with uncertain missing modalities. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1th International Joint Conference on Natural Language Processing (V (Volume 1: Long Papers), pp. 2608 26188 2021. 2
13

---

Published as a conference paper at ICLR 2025
A PROPOSED SOLUTION: EXTENDED DISCUSSION
In Section 4, we proposed our novel adaptation strategy; MiDI. First, note that our intuition of minimizing the mutual information between the output of the network and the modality source comes from the following observation: Let X and Y be two random variables, then
If MI Y) = 0 X and Y are independent.
Pxx (2, (x(x)
Proof: MI(X,Y) Σ Σ
Pxy Pxy log
x yEY
If (I(XX)) + yx
That is. minimizing the mutual information between the output prediction of fe and the available modality should make the adapted network robust against missing modality. Second, MiDI adapts the pretrained model fo on the unlabeled data revealed from the stream only if the revealed data is with complete modality. This since: (1) the KL-ddvergeecc loss only applies on data with full modality and (2) the information minimization loss only operates under complete modality. To wit, let x; be revealed with m A. Then the estimate of our LMi will be
LMI
falai; M = A) log (Jō(215 M = A))
K
Σ
Pala; M A) log (fi(zz; M = A))
This is since E,, will be estimated with single point at A. Thus, MiDI adapts the parameters only when S reveals data with complete modality at test time.
B
EXPERIMENTS
B.1 IMPLEMENTATION DETAILS
In Section 5. we detailed our experimental setup. Here, and for reproducibility. we provide the implementation details for MiDI and the baselines. Note that for all considered adaptation methods, we follow the standard practice in the test-time adaptation literature (Wang et al., 2020; Liang al., 2020; Niu al., 2022; Niu14 al., 2023) and only adapt the learnable parameters of the normalization layers. We freeze the rest of the model parameters during the update step. Further, for MiDI, we balance the mutual information loss and the self-distillation loss through:
EMIDI = CMI + An
Note that A A₂2 3 for all our experiments. These hyperparameters were determined through grid search to identify the optimal settings for the task. Further, we conduct the adaptation step with an SGD (Robbins & Monro, 1951) step with a learning rate of 25 10 and momentum of 0.9, following (Niu14 et al., 2023; Niu et al., 2022; Wang et al., 2020).
Regarding the considered test-time adaptation baselines, we considered the entropy minimization approach known as Tent (Wang et al., 2020), its improved version equipped with data-selection process ETA (Niu et al., 2022), and the information maximization Shot (Liang et al., 2020). We followed the official implementation of each method and used the recommended hyperparameters.
Fach experiment was run using one V100 GPU. We repeat experiments times with different seeds and report the average accuracy.
B.2 BASELINES UNDER LONG TERM ADAPTATION
We extend our comparison against the considered baselines under the Long Term Adaptation (LTA) setup. We replicate experimental setup in Section 5.3 and compare MiDI against SHOT, Tent, and
14

---

Published as a conference paper at ICLR 2025
Table 7: Adaptation at Test-time under Long-term Adaptation. We showcase the results of MiDL under the assumption that the data stream is very long. We use unlabeled data to simulate longer stream and report results on the validation set of each dataset. Our MiDI benefits from long-term adaptation. Especially at higher missing rates (>75%).
1 - PAV
Sepic-Sondee (%)
Epic Kitchens (%)
Model
25
75
100
0
25
50
100
Unimodal
41.4
41.4
41.4
41.4
41.4
40.0
40.0
40.0
40.0
40.0
Baseline
55.1
45.6
37.1
28.3
19.5
63.9
55.5
46.8
37.9
29.5
+Shot LTA
55.0
45.6
37.2
28.7
20.3
63.8
56.0
48.2
41.0
34.4
+ +Tent LTA
54.5
44.6
35.5
26.1
17.7
62.7
53.5
40.0
25.8
12.3
+ETA LTA
55.0
45.5
37.0
28.2
19.5
60.9
48.8
33.0
19.3
7.5
+MiDI LTA (ours)
54.9
46.8
39.5
32.6
26.0
63.7
58.4
52.4
46.7
41.4
Table 8: Adattation at Test-time Updating all parameters. We show the results when we unfreeze all network parameters, only the normalization layer. We observe that there is no significant difference when compared to updating only the normalization layers.
1 1 PAV PAV
Eppic-Kithher (%)
Model
0
25
50
75
100
Unimodal
40.0
40.0
40.0
40.0
40.0
Baseline
63.9
55.5
46.8
37.9
29.5
+MiDI (all parameters)
63.6
58.4
52.4
46.3
29.5
+MiDI (norm layers)
63.8
58.4
52.1
44.9
29.5
+MiDI LTA (all parameters)
63.6
58.3
52.5
47.1
42.0
* MiDI LTA (norm layers)
63.8
58.4
52.4
46.7
41.4
ETA. We report the results in Table for the MBT architecture. We observe consistent findings in Section 5.3, where naive adaptation baselines do not benefit from this long-term adaptation. Further, we find that MiDI is the only adaptation method that provides further performance gains under this LTA setup.
Table 9: MiDL Performance with Mixed Modalities Setup. We present results similar to those in Table for Epic Sounds, but this time under mixed missing modalities test time. Our results demonstrate that MiDL continues to enhance the base model's performance even in the challenging scenario where any modality may be absent.
PAV
SpiccSSounds (%)
Model
Baseline
55.1
44.0
39,5
34.1
+MiDI (ours)
55.1
50.0
45.0
40.3
34.1
+ +MDI LTA (ours)
55.0
50.3
45,4
42.1
37.4
B.3 ADAPTING MORE LAYERS
At last, we extend our analysis on MiDI to include the effect of adapting more parameters. In particular, compare adapting only the learnable parameters of the normalization layers against adapting the whole network parameters. We report the result in Table on Epic-Kitchens. Kitchens. We observe that adapting all network parameters with MiDI results in minor performance gain. For instance, under the 75% missing rate, adapting all parameters improves over adapting only the normalization layers by 0.4% under the LTA setup and by 1.4% with the test-time adaptation of MiDI. We note
15

---

Published as a conference paper at ICLR 2025
Table 10: KL loss on each prediction. We apply KL loss the prediction of each modality, As the audiovisual predictions are derived from the individual modality predictions, there is no much difference in the performance.
Pav
SpiccSSunds (%)
(%)
Model
0
25
50
75
100
25
100
Unimodal
41.4
41.4
41.4
41.4
41.4
40.0
40.0
40.0
40.0
40.0
Baseline
55.1
45.6
37.1
28.3
19.5
63.9
55.5
46.8
37.9
29.5
+MiDI (ours)
55.0
46.8
38.8
29.8
19.5
63.7
58.4
52.4
46.4
29.5
+MiDI (KL. on each modality)
55.0
46.8
38.9
30.0
19.5
63.5
57.9
52.1
46.4
29.5
Table 11: Combating missing modalities at Test-time. We present the extended results of Table 1 that include the standard deviation of each of the results. Our proposed MiDI is effective at combating missing modalities at test time, outperforming all presented TTA baselines by convincing margins over several runs.
Epic-Soundh (%)
Epic Kitchens (%)
Model
75
100
50
100
Unimodal
41.4
Baseline
55.1
45.6
37.1
28.3
19,5
63.9
55.5
46.8
37.9
29.5
+Shot
55.0#004
45.6:u02
37.1
28.5=00
20.0=0.0
63,9=004
55.920.5
47.9
40.6
34.3:01
54.8+0.4
45.0=008
35,9#006
26.5:004
17.8=005
54.0=0.15
39.240.25
9.9-022
+ETA
55.110.02
45.620.02
37.11001
28.3
19.5
63.5=0.04
51,320.7
33.740.26
20.61024
7.9=027
+MiDI (ours)
55.0+0.10
46.8
38.8 11 29.8=0007
19.5
63.7=0008
58.40044
$2.4
46.4
29.5
here that this comes computational expense it faster and more efficient adapt only the normalization layers.
B.4 MIXED MODALITIES
In this setup also Pav (0.0, 0.25, 0.5, 0.75, 1.0), but also PA Pv 0.5 (1 PAv). Thus, for each missing rate (1 PAv), exactly half of the modal-incomplete samples have missing audio, and the other half have missing video. The results of this setup are shown in Table We observe that MiDI consistently improves the baseline performance for all missing rates. Furthermore, when presented with long stream (LTA), MiDI benefits by further improving the accuracy.
B.5
KL LOSS ON ALL PREDICTIONS
One might suggest upplying KL loss to both the individual audio and video predictions, as well as the combined audiovisual predictions. We present the results of this experiment in Table 10. We found that applying KI. loss individually to each modality produced results very similar to applying it to the combined audiovisual predictions. This is because the combined predictions essentially average the individual modality predictions, making the effect of applying the loss individually or in combination nearly equivalent.
C LIMITATIONS
As we mentioned in Section 6.5, MiDI requires three forward passes of the model over a single instance, which can be implemented efficiently by parallelizing the pass the GPU. However, it does require more FLOPs than a method without adaptation. Additionally, MiDI can only adapt to modal-complete instances. Finally, our experiments are limited to audiovisual egocentric datasets. Although we designed MiDI without these constrains in mind, the validation of our method was only made under this setup. Thus, results other multimodal datasets with other modalities and other data sources need further validation. However, we believe that MiDI would still work in other scenarios.
16

---

Published as a conference paper at ICLR 2025
Positive Cases (Model Successes)
MISSING
NISSING
MISSING
Ground Truth: chair
No Adaptation Predlctton:cuphoard
MIDI Prediction:
chair
(a) Positive Case
MISSING
HISSING
MibsINo
MisSiNo
Ground Truth: mix
No Adaptation Prediction:
wash
Miol Prediction: mix
(b) Positive Case Negative Cases (Model Failures)
MISSINO
MESSINO
Ground Truth: turn-on
No Adaptation Pradiction: turn-on
MiDI Prediction:
wash
(c) Negative Case 1
Ground Truth: open
No Adaptation Pradiction: open
MiDI Prediction:
close
(d) Negative Case 2
Figure 3: Qualitative analysis of MiDI's adaptation performance Epic-Kitchens. The top
two subfigures highlight positive cases where MiDI successfully adapts to predict the correct label (marked in green). Conversely, the bottom subfigures illustrate negative cases (marked in red) where adaptation introduces emros..
17

---

Published as a conference paper at ICLR 2025
Positive Cases (Model Successes)
MISSING
Ground Truth: chop
No Adaptation Prediction:
vipe
MIDI Prediction:
chop
(a) Positive Case
MISSING
Ground Truth: metal-only collision
No Adaptation Prediction:
whisk
MiDl Prediction:
metal-only collision
(b) Positive Case
Negative Cases (Model Failures)
MISSING
Ground Truth: wips
No Adaptation Pradiction:
wipe
MiDI Pradiction:
water
(c) Negative Case
MISSING
Ground Truth: wips
No Adaptation Prodiction: wipe
MiDI Pradiction:
metal-only collision
(d) Negative Case 2
Figure 4: Qualitative analysis of MiDI's adaptation performance Epic-Sounds. The top two subfigures highlight positive cases where MiDI successfully adapts to predict the correct label (marked in green). Conversely, the bottom subfigures illustrate negative cases (marked in red) where adaptation introduces enrors.
D QUALITATIVE RESULTS
Figure presents a qualitative analysis of our method's test-time adaptation for audiovisual models on Epic-Kin Kitchen,, The top subfigures (Positive Cases) highlight successful adaptations. where the
18

---

Published as a conference paper at ICLR 2025
model accurately compensates for the missing modality to predict the correct label. For example, in Positive Case 1, the model initially predicts "cupboard" without adaptation but successfully adapts to the correct label "chair" by leveraging the sound cues. This demonstrates how MiDI effectively uses auditory signals to distinguish the distinct sound of a chair. Similarly, in Positive Case 3, the model transitions from incorrect prediction of "wash" the correct action "mix."
In contrast, the bottom subfigures (Negative Cases) illustrate instances where adaptation introduces errors. For instance, Negative Case the model changes its correct prediction of "turn-on" (without adaptation) to an incorrect "wash," and in Negative Case 2, the method erroneously adapts from "open" (correct without adaptation) to "close."
Likewise, Figure 4 showcases the qualitative performance of our test-time adaptation method in scenarios where the audio modality missing. using Epic-Sounds. The top subfigures (Positive Cases) demonstrate successful adaptations that align the predictions with ground truth despite the missing audio. For example, in Positive Case 1, the model refines its initial prediction from "wipe" (without adaptation) to the correct label "chop." effectively utilizing visual cues. Similarly, in Positive Case 3, the prediction adapted from "whisk" to the correct "metal-only collision," showcasing MiDI's capacity to mitigate the absence of sound.
However, the bottom subfigures (Negative Cases) reveal errors introduced by adaptation. In Negative Case 1, the model changes correct "wipe" prediction to incorrect "water," potentially due to the visually confusing presence sink without audio context. In Negative Case 2, the method misclassifies "wipe" as "metal-only collision," " likely influenced by visible metallic objects, making a plausible yet incorrect adaptation.
Overall, Figures 3 and illustrate both the strengths and limitations of our test-time adaptation method. While MiDI frequently improves prediction accuracy, as evidenced by our quantitative results, it occasionally induces misclassifications. To provide balanced perspective, we included an equal number of success and failure cases. These failure cases offer valuable insights, paving the way for further refinements and future research.
E ADDITIONAL RESULTS ON EGO4D
We evaluated MiDL's performance on the Ego4D-AR dataset Ramazanova et al. (2024), derived from Ego4D (Grauman et. al., 2022), assess its generalizability and robustness under varying levels of missing audio. Ego4D-AR encompasses diverse daily activities and naturally includes instances of missing audio. As shown in Table 12, MiDL consistently surpasses baseline methods across 50%, and 75% missing audio rates
MiDL demonstrates significant advantages, achieving 23.41% accuracy 75% missing rate, compared to 21. 21.46%, 15.92% and 22.06% for Baseline, TENT, and SHOT, respectively. These results highlight MiDL's ability effectively handle missing modality scenarios, adapting to challenging conditions where conventional approaches struggle.
By offering greater resilience incomplete noisy data, MiDL establishes itself a robust solution for multimodal learning in egocentric video applications. These findings underscore its potential to advance state-of the-art methods, particularly real-world settings characterized by data sparsity or inconsistency.
Table 12: MiDL Performance on Ego4D-AR with Missing Audio. Performance comparison at various missing rates of audio (1 PAv).
1 PAV (%)
Ego4D ARR (%)
Model
50
75
100
Baseline
26.21
21.46
16.58
TENT
23.29
15.92
9.25
SHOT
26.56
22.06
18.29
MIDL (ours)
27.13
23.41
16.58
19

---

Published as a conference paper at ICLR 2025
CONTRIBUTION STATEMENT
We highlight the contribution each author in this project. Merey Ramazanova coordinated and refined the project's focus on missing modality challenges in egocentric videos. She led the imple- mentation and experimental design, including MiDI, baseline models, all multimodal setups, and ablation studies. Additionally, she managed data preparation across all datasets and co-designed the evaluation protocols. She also proposed the oot-of-Domain Adaptation setup using Ego4D.
Alejandro Pardo refined the code for large-scale experimentation, assisted in the baseline implementa- tion, conducted the experiments, and explored various ablations and parameters for each method. He played significant role in writing and presenting the work's experimental section and the qualitative results.
Bernard Ghanem guided the project from the beginning, helped shape the initial ideas, and signifi- cantly contributed to the design of the experiments and the project's overall development. He also proofread and polished the manuscript.
Motasem Alfarra initiated the project by formulating the missing modality challenge test-time adaptation problem, proposed the MiDI objective function, co-designed the evaluation protocols, led the writing efforts, and advised the implementation of MiDI along with the other baselines.
20


Reviews:
[{"review_id": "rnHnCn6uH3", "reviewer": "Reviewer_6iPc", "summary": "To tackle the issue of modality missing in real-time tasks, this framework offers an online self-supervised learning method called MiDl. MiDl uses mutual information and KL divergence as loss functions to optimize the model in real time, enabling the baseline model to better handle inputs with missing modalities.", "questions": "1. If this approach faces extreme examples, such as a video showing a calm street while the audio is an explosion, will this model mislead the baseline model into the wrong direction?\n2. You might consider adding extra blocks to the model, so that if updates are needed, only the added portions need to be updated. Alternatively, updating part of the model's structure could prevent the significant latency introduced by updating the entire system.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper redefines the modality missing problem as a test-time adaptation (TTA) issue, emphasizing the challenges of modality absence faced in online multimodal tasks. This is indeed an urgent problem that needs to be addressed for many online multimodal tasks.\n\nThe proposed MiDl method effectively enhances the baseline model's ability to handle missing modalities, serving as a solution for the modality missing problem in multimodal online tasks. This approach can act as a supplement when facing modality absence in such tasks. For instance, if modalities are functioning normally, this pipeline may not be used; however, when a modality is missing, the proposed solution can improve the baseline model's capability to handle the missing modalities. Additionally, normal inputs and prediction results can serve as supplementary information when modalities are insufficient.\n\nThe experiments presented in the paper are comprehensive, demonstrating that the \nmethod is independent of modality selection, baseline models, and model frameworks, thereby proving the robustness of the proposed solution.", "weaknesses": "1. \"First, the prediction of should be invariant to the modality source . Ideally, f_{\\theta} should output the same prediction under both complete and incomplete modality, hence satisfying the following equality: (i)...,\" The underlying assumption of the approach is controversial. The task will degenerate into a modality distillation problem if this assumption holds,. Is there a more reasonable way to phrase this?\n2. Implementing this method in real-world production could introduce significant computational overhead and latency. Normal models can be accelerated through techniques like compression and distillation, but this approach involves updating model weights, requiring the retention of the complete model, making it difficult to deploy directly in practice.\n3. Could you include experiments demonstrating the approach's decision-making in more complex online scenarios? The experiments provided in the paper do not represent the best use case for this method; its most suitable application is in online scenarios, so experiments in these contexts would better support the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8lgtN8uxal", "reviewer": "Reviewer_CsYs", "summary": "This paper tackles on missing modalities in egocentric videos without the need to retrain models by formulating this challenge as a test-time adaptation task. The authors proposed MiDl which minimizes the mutual information between prediction and the modality, with the incorporation of self-distillation to maintain performance when all modalities are available. The author benchmarked several methods under such problem formulation, demonstrating a descent performance when part of the modality are missing in two egocentric video datasets.", "questions": "Please refer to the Weakenesses section. I highly encourage the authors to directly include the code for the verification of reproducibility.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is interesting and easy to follow. \n- The formulation of the test-time adaptation for tackling missing modality without the need for model retraining is indeed novel and can be foreseen to be applied to various applications which contains multi-modal information.\n- Although the method itself is not complex and consists of components already used for various tasks for multi-modal learning and egocentric video analysis, they are leveraged in MiDl with strong motivation which are intuitive and reasonable. The extended discussion also offers a deeper understanding over the formulation of MiDl. This method could prove to be a good starting point for subsequent discussions in the research community.\n- The authors also provide a comprehensive analysis over the performance of MiDl in the formulated task, and also benchmarked previous methods such as SHOT, TENT under the same setting, which also provides a further insight into the challenges and possible methods to further tackle the task.\n\nIn general, this paper is relatively well presented with a simple yet highly motivated method for an interesting formulation of a realistic challenge.", "weaknesses": "There are a few minor concerns remaining in the paper, mainly on the clarity and possible extension in discussion of the proposed method. I would like the authors to consider the following concerns if possible:\n1. On Page 4, Line 197-198, the author states that \"$f_\\theta$ should retain high performance in predicting data with complete modality, which is generally satisfied for $f_{\\theta_0}$\". Does this imply that the non-adapted pretrained model must be pretrained with all modalities available? What if the pretrained model is only trained with a single modality (e.g., only with visual information without the audio information which is rather common in video models)?\n2. It is observed that there is a large drop when $1-p_{AV}=100$, where none of the data contain both modalities. What would be a possible approach to mitigate this drop in performance. It is observed that the drop for MiDl is significantly more severe than that of SHOT.\n3. The current method only touches upon the case for two modalities (audio and video), is it expandable towards more modalities. Also, are there limitations for the possible types of modalities or it can be any modalities as long as they are obtained from the same set of data?\n4. The experiments are performed for each dataset with a drop in the primary modality, what would be the result if the secondary modality is dropped with the same probability?\n5. Lastly, the code is currently NOT available, which means that the reproducibility of the result is not verified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WGevhaOgup", "reviewer": "Reviewer_M6ux", "summary": "This paper presents a novel approach to handling missing modalities in multimodal learning using test-time adaptation. The method, MiDl, shows promising results across the EPIC kitchen and EPIC sounds datasets, and the method is motivated by the theoretical intuition of minimizing the mutual information between the predicted and available modality. The authors also provide some interesting analysis of the model through long-term adaptation, out-of-distribution warmup, and various ablation experiments. \n\nThis review follows the sections of the paper.", "questions": "While the technical aspects and experimental results are generally strong, there are areas for improvement in the motivation, clarity of presentation, and some experimental details. \n\nI presented many questions and suggestions in the weaknesses suggestions. In particular, I would suggest the authors focus on the concerns about the motivation and the experiments aligning with that motivation. My comments regarding notation and small fixes are merely suggestions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Introduction:\n1. The second and third paragraphs effectively identify the gap in the literature and provide a robust overview of the proposed solution.\n\nRelated Works:\n\n2. Related works is concise and relevant\n\nMissing Modality as Test Time Adaptation:\n\n3. This section is well-written and easily comprehensible.\n\nExperiments:\n\n4. It is commendable that the experiments were repeated 5 times with reported standard deviations in Table 11; the results appear experimentally robust and convincing.\n5. The Ego4d Warmup result on 100% missing rate in the original distribution is an interesting finding with potentially strong applications.\n\nAnalysis on MiDL:\n\n6. The Components of MiDl section is strong, and the ablations are empirically sound.", "weaknesses": "Introduction:\n1. The motivation presented in the first paragraph is weak. For instance, in Line 40, could you provide an example or application where inference must occur on a redacted modality? I can think of the application of blurring faces in images or deleting private details in medical records, but it's unclear when only one modality would be completely removed for privacy reasons while the other remains intact for the same data instance. Additionally, the relevance of using cheaper modalities to missing modalities is not apparent (line 40). It would be particularly convincing if the motivation aligned with the tested dataset. For example, if using Epic Kitchens, perhaps a scenario involving a humanoid with a malfunctioning audio sensor, or smart glasses with an obscured camera due to steam or food spillage could be considered.\n2. Contribution (3) appears to be describing MiDl, which is already covered in contribution (2). I would recommend reassessing what could constitute a third distinct contribution from your work.\n3.Figure 1 requires improvement. The concept of a \"potential performance trajectory\" needs clarification - is this your hypothesis? This graphic would be more persuasive if it depicted your actual no-adaptation baseline and your TTA method. The purpose of the black line in the middle of the graph is unclear.\n\nProposed Solution:\n\n4. The notation in eq (1) lacks precision. It is not evident that f(x;m) and m are random variables. The output of f is a distribution. Are you considering this as a random variable, with the value being the indices and the probability of those values the logits? Consider introducing a random variable Y ~ f(x;m). Also, consider using capital letter notation (e.g. \"M\") for the random variables. Furthermore, how can you evaluate the KL if x ~ S only has the modality m, not AV? Later in this section, it becomes apparent that you only update the model on complete instances. This limitation/assumption should be made clearer in the introduction or Takeaways subsection. This method would only be applicable for testing data that includes some multimodal instances.\n5. At last line of page 4, is $x_t$ a single sample? Do you mean samples $x_0 \\dots x_t$?\n\nExperiments:\n\n6. Additional details could enhance the reproducibility of this work. Was any hyperparameter tuning conducted for MiDl? Section B.1 mentions the recommended hyperparameters for the baseline but doesn't specify how they were determined for MiDl. Moreover, what were the proportions of the train/val/test split?\n7. In section 5.3 LTA: You allow the model to retrain on some of the unlabeled training data. Why not gather $S_{in}$ from the validation set? In this setting, is MiDl trained on $D \\ S_{in}$? Or is $S_{in}$ still included in the labeled dataset before test time and then used without labels during test time?\n8. Can this method be applied to instances where either modality is missing, e.g., P = {.3,.3,.3}? It would be great to see results for experiment with such a ratio. Currently, it may be the case that the model learns to leverage ONLY the modality that is consistently present in both the complete modality test case and the missing modality test case. In this scenario, would a given unimodal model for the always-present modality perform optimally? Table 1 could be improved by clarifying what is meant by \"Unimodal\" and why it is only present at the 50% missing rate. For Epic Sounds, is the unimodal the always-present modality (video)?\n\nAnalysis on MiDL:\n\n9. Both architecture choices are transformer-based. It would have been more convincing to see a greater diversity of architectures (such as a convolution backbone). Instead of presenting different missing rates as columns in Table 3, it would have been preferable to see different architectures/methods as the columns with a fixed missing rate (perhaps 50%).\n10. Given that the main motivation was to avoid retraining an existing method on a large dataset to perform missing modality adaptation, the results would have been more convincing if the authors had either used an existing model+dataset and just performed adaptation, as opposed to training from scratch an existing method. Alternatively, they could have tested with a very large dataset that was computationally expensive. The omnivore pretraining test is good. Did you train the model from scratch on your dataset or use an existing model and apply MiDl?\n11. In Table 6, shouldn't 55.2 in the Dl column be bolded?\n12. I thought the motivation was that retraining on the train set is computationally expensive, and TTA will prevent that? It's good that you acknowledge the computational requirements of MiDl, but then in the abstract, you shouldn't state: \"Current methods, while effective, often necessitate retraining the model... making them computationally intensive.\" Alternatively, compare your inference computation here with the amount of computation required to retrain training data (to get sufficient performance).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xVqH142uXb", "reviewer": "Reviewer_wPGf", "summary": "In this work, the authors focus on an important task which is test time adaptation for egocentric video action recognition under missing modality. The authors validate existing work of TTA on this new task and propose a new method MiD1 to enhance the robustness of the learned features. The performance of the proposed method is evaluated on the EpicKitchen sound and video dataset", "questions": "1. Could the authors include comparisons with other approaches specifically addressing the missing modality issue, such as those proposed by Dai et al. (2024), Lee et al. (2019), and Wang et al. (2023)?\n\n2. Given the importance of this task, would the authors consider expanding the benchmark by including more test-time adaptation (TTA) approaches, such as Chen et al. (2022), Wang et al. (2020), Yuan et al. (2023), and Niu et al. (2022), and analyzing performance across different clusters of approaches? Including missing modality approaches as baselines may also strengthen the benchmark.\n\n3. Could the authors provide qualitative results comparing their approach to the baseline, along with a failure case analysis to offer insights into scenarios where the method may fall short?\n\n4. Has the generalizability of the proposed approach been tested on datasets beyond EPIC Kitchen? If not, would the authors consider verifying the performance on additional datasets?\n\n5. Could the authors use TSNE visualization on the latent space to illustrate how the proposed supervision affects feature learning? Specifically, visualizing changes over different epochs in comparison to the baseline might provide additional insights.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Missing modality issue is important for test time adaptation of ego centric action recognition. This task will contribute to the community.\n\n2. Method section is clearly written and easy to follow.\n\n3. Compared with the baseline, the proposed approach show good performance on this new task.", "weaknesses": "1. Lack of the comparison with other approaches specifically targeted at missing modality issue. \n\na. Dai, Y., Chen, H., Du, J., Wang, R., Chen, S., Wang, H., & Lee, C. H. (2024). A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 27445-27455).\n\nb. Lee, H. C., Lin, C. Y., Hsu, P. C., & Hsu, W. H. (2019, May). Audio feature generation for missing modality problem in video action recognition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 3956-3960). IEEE.\n\nc. Wang, H., Chen, Y., Ma, C., Avery, J., Hull, L., & Carneiro, G. (2023). Multi-modal learning with missing modality via shared-specific feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15878-15887).\n\n\n2. The authors are suggested to enlarge the benchmarks. I aggree that this task is an important task, however the experiments are limited in this paper which will be harmful to its soundness. The authors could enrich the benchmark using more existing TTA approaches, e.g., d,e,f, and g, and try to provide an analysis on the performance on different cluster of approaches. Missing modality works can also serve as good baselines to enrich the benchmark.\n\nd. Chen, D., Wang, D., Darrell, T., & Ebrahimi, S. (2022). Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 295-305).\n\ne. Wang, D., Shelhamer, E., Liu, S., Olshausen, B., & Darrell, T. (2020). Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726.\n\nf. Yuan, L., Xie, B., & Li, S. (2023). Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15922-15932).\n\ng. Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P., & Tan, M. (2022, June). Efficient test-time model adaptation without forgetting. In International conference on machine learning (pp. 16888-16905). PMLR.\n\n3. No qualitative reustls. The authors are suggested to provide some qualitative results when comparing their approach with the baseline approach. Some failure case analysis will be helpful.\n\n4. The performance of the proposed approach is only verified on EPIC Kitchen, generaliyability to other dataset can be an issue.\n\n5. TSNE visualization on the latent space will be helpful to see how the proposed supervision help during the feature learning procedure. The authors could visualize the changes for different epoches compared with its baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)