Test-Time Adaptation for Combating Missing Modalities in Egocentric Videos

## Abstract

Abstract Understanding videos that contain multiple modalities is crucial, especially in egocentric videos, where combining various sensory inputs significantly improves tasks like action recognition and moment localization.
However, real-world applications often face challenges with incomplete modalities due to privacy concerns, efficiency needs, or hardware issues.
Current methods, while effective, often necessitate retraining the model entirely to handle missing modalities, making them computationally intensive, particularly with large training datasets.
In this study, we propose a novel approach to address this issue at test time without requiring retraining.
We frame the problem as a test-time adaptation task, where the model adjusts to the available unlabeled data at test time.
Our method, MiDl (Mutual information with self-Distillation), encourages the model to be insensitive to the specific modality source present during testing by minimizing the mutual information between the prediction and the available modality.
Additionally, we incorporate self-distillation to maintain the model’s original performance when both modalities are available.
MiDl represents the first self-supervised, online solution for handling missing modalities exclusively at test time.
Through experiments with various pretrained models and datasets, MiDl demonstrates substantial performance improvement without the need for retraining.

###### Keywords:

## 1 Introduction

Understanding multi-modal data has emerged as a pivotal challenge in various domains, including foundational model construction [], emotion recognition [], and analysis of egocentric videos for tasks [, , ] like recognition [, , ] and localization [, ].
Recent efforts focused on crafting models harnessing data diverse modalities showing performance gains in multiple tasks [].
However, a critical limitation arises: these models presuppose complete modality availability at test-time, which diverges from real-world scenarios where data modalities may be incomplete [].
For instance, in real-time prediction using wearable devices, portions of recordings might be redacted to safeguard privacy, or cost constraints may necessitate using cheaper modalities like audio or IMU [, ].
Consequently, models designed under this assumption demonstrate significant performance degradation in the face of missing modalities, sometimes even underperforming unimodal counterparts trained with a single modality [].

The challenge of addressing missing modalities gained significant attention from researchers recently, where various strategies have been proposed to mitigate this issue [, , , ].
Some works have explored architectural modifications to fuse information from different modalities effectively [].
Additionally, other approaches have focused on designing effective regularizers to boost model performance when confronted with missing modalities [].
More recently, a promising direction has emerged, wherein transformer models have been augmented with learnable tokens during training [, ]. These tokens serve to compensate for missing information at test time, significantly enhancing model robustness against missing modality [].
Despite these advancements, a common drawback persists: all existing approaches necessitate expensive retraining of the multi-modal model, rendering pretrained models obsolete.
This poses a substantial challenge, particularly in applications with extensive training data, where the retraining process is prohibitively expensive, making the aforementioned approaches impractical.
Consequently, a fundamental question arises:

*
Can we develop methods to address missing modalities at test time without imposing retraining requirements?
*

Figure: Figure 1: Test-Time Adaptation for missing modalities. The concept of test-time adaptation in the presence of missing data modalities focuses on a system where a stream of multi-modal data is input, potentially lacking one or more modalities (represented by audio and visual symbols). Without adaptation, the pretrained model $f_{\theta_{0}}$ processes the samples but may not accurately predict labels due to incomplete data. With test-time adaptation, the model is dynamically adjusted using the adaptation method $g$, resulting in an adapted model $f_{\theta_{t}}$, which is designed to handle the missing modalities and improve over time. The graph on the right illustrates the potential performance trajectory of models with and without test-time adaptation, showing the efficacy of adaptation in maintaining higher performance levels despite the variability in data modality availability.
Refer to caption: extracted/2404.15161v1/figures/pull.png

In this work, we take the initial step of framing the missing modality problem as a test-time adaptation problem [, , ].
Specifically, we aim to establish a new approach wherein pretrained models undergo adaptation at test time to optimize their performance in the presence of missing modalities.
Our formulation assumes a scenario where a continuous stream of unlabeled data is fed into the pretrained model during testing, with some segments missing certain modalities.
The objective is to devise an adaptation algorithm capable of refining the model’s predictions under missing modality in real-time settings (refer to Figure 1).
Based on this formulation, we first evaluate existing methodologies from the test-time adaptation literature and demonstrate their limited efficacy in addressing this specific challenge.
Subsequently, we introduce a novel test-time adaptation technique explicitly tailored to tackle the missing modality problem.
Our method revolves around incentivizing the output of the pretrained model to remain invariant to the modality source present during testing.
To achieve this, we propose minimizing the mutual information between the model’s output and the modality type of the incoming unlabeled data at test time in a self-supervised manner.
Moreover, to ensure the preservation of model performance under a complete modality setup, we integrate our approach with a self-distillation mechanism.
Notably, our method, termed MiDl (Mutual information with Self-Distillation minimization) is theoretically motivated and agnostic to the choice of pretrained model architecture, the dataset, and the specific type of missing modality encountered during testing.

We summarize our contributions in three-fold:

- 1.
We redefine the missing modality problem as a test-time adaptation challenge, pioneering a novel approach where pretrained models are adapted in real-time to optimize performance in the face of missing modalities.
We evaluate the effectiveness of the current adaptation method under this challenging problem.
- 2.
We introduce MiDl (Mutual information with Self-Distillation minimization), a test-time adaptation method explicitly tailored to combat the missing modality problem. MiDl incentivizes model outputs to remain invariant to modality sources during testing, effectively enhancing robustness.
- 3.
Our approach offers versatility, being agnostic to factors such as pretrained model architecture, training dataset, and the specific type of missing modality encountered during testing, thus providing a comprehensive solution applicable across diverse scenarios.
When combining pretrained models with MiDl, a significant performance gain is attained (6% on epic sounds and 11% on Epic Kitchens datasets).

Code to reproduce all our experimental results can be found [in this repo](link).

## 2 Related Wrok

#### Missing Modalities in Multimodal Datasets.

Several works addressed the problem of missing modality in the multimodal datasets [, , , , ].
Most methods addressing the missing modality problem assume full access to the source (training) data. Some works are based on the assumption that the training data is modal-complete, and the goal is to train a model robust to the missing inputs at test time [, ]. Other works tackle the modality distillation task, where the training data is multimodal, but only one modality is used at test time [, ]. Few works assume the modalities could be missing at train and test times and attempt to train a robust network [, ].
In our work, we explore a more realistic scenario where we might not have access to the training data or network re-training is not feasible.
We formulate this setup as a test-time adaptation problem, where the model is experiencing a distribution shift caused by the unavailability of some modalities at test time.
Further, we propose the first test-time adaptation algorithm tailored to combat the missing modality challenge at test-time.

#### Test-Time Adaptation.

Test-time Adaptation (TTA) attempts to combat performance gaps that pretrained models suffer from when exposed to distribution shifts at test-time [, ].
This is usually attained through
modifying the model’s parameters [] or its input [] by using the incoming unlabeled data at test-time.
TTA methods are practical, as they avoid assumptions on the training phase of a given model [].
The first of these approaches adjusts the statistics of the Batch Normalization (BN) layers [].
This was followed by more powerful adaptation methods that involved self-supervised objective functions such as entropy minimization [, , ], information maximization [], and teacher-student approaches [].
While such TTA methods made significant progress towards combating distribution shifts at test-time, they solely focused on simple covariate shifts such as changes in weather conditions or pixel illumination [].
In this work, we extend the problem formulation of test-time adaptation to a very practical and realistic domain shift: missing modality.
In particular, we adapt the stream setting of online test-time adaptation [] to formulate the missing modality problem along with the corresponding evaluation protocol.
Building on this novel view of the missing modality problem, we analyze the current state-of-the-art TTA methods where we show their limited impact.
Further, we propose a novel TTA method tailored to combat the missing modality problem.

## 3 Missing Modality as Test Time Adaptation

In this section, we formulate the missing modality problem as a test time adaptation problem 3.1.
We then outline the evaluation protocol of a given adaptation method in Section 3.2

### 3.1 Problem Formulation

In this work, we focus on the recognition problem.
Let $f_{\theta}:\mathbb{R}^{d}\rightarrow\mathcal{P}(\mathcal{Y})$ be a classifier that maps a given input $x\in\mathbb{R}^{d}$ into the probability simplex $\mathcal{P}(\mathcal{Y})$ over the set of labels $\mathcal{Y}=\{1,2,\dots,K\}$(^1^11*e.g.* the output after a softmax layer.).
In this work, we assume that an input $x$ is a multi-modal input.
However, in many realistic applications, the input provided to the model might have missing modalities, thus containing either audio only, visual only, or audio-visual information.
Let $m\in\{A,V,AV\}$ denote the type of available modality for a given input $x$ corresponding to audio only, visual only, audio-visual, respectively.
For simplicity, we fix the dimensionality of the input $x$ by replacing the missing modality part with either zeros or random data.
Further, we assume that $f_{\theta}$ is pretrained on a training set $\mathcal{D}$.
In this work, we make no assumptions on $f_{\theta}$ (*i.e.* choice of architecture), the dataset $\mathcal{D}$, nor the way the training process is done.

At test time, $f_{\theta}$ is presented with a stream of *unlabeled* data $\mathcal{S}$ with possibly missing modalities.
The likelihood with which a certain modality appears in data revealed from $\mathcal{S}$ is characterized by a probability mass function; denoted by $\mathbb{P}_{\mathcal{S}}(M=m)$.
For example, if $\mathbb{P}_{\mathcal{S}}(M=V)=0.5$ and $\mathbb{P}_{\mathcal{S}}(M=AV)=0.5$, then the audio missing rate in the test stream is $50\%$.
This is since 50% of the data comes as video only, thus missing its audio modality.
Let $p_{m}=\mathbb{P}_{\mathcal{S}}(M=m)$, then the missing rate of different modalities can be equivalently characterized with $P=\{p_{A},p_{V},p_{AV}\}$.
Thus, for a stream with 25% missing video, $P=\{0.25,0.0,0.75\}$ (*i.e.* the stream will reveal data with 25% probability of having only audio and 75% probability of revealing both modalities).
According to this characterization, one can define the missing rate of at least one modality as $1-p_{AV}$ with $p_{A}$ being the rate of missing video and $p_{V}$ being the audio missing rate.
Next, we discuss the online evaluation protocol of $f_{\theta}$ under the stream $\mathcal{S}$ of unlabeled data.

### 3.2 Evaluation Protocol

Given our formulation of missing modality in Section 3.1, we are now ready to outline the evaluation protocol.
Note that an adaptation method is a function $g(\theta)$ that sequentially adapts the model’s parameters $\theta$ to enhance the performance under the missing modality setup.
Formally and following the online learning literature [, ], we simulate the interaction between the stream $\mathcal{S}$ characterized by $P$ and the TTA method $g$, at each time step $t\in\{0,1,\dots,\infty\}$, as follows:

where
$f_{\theta_{0}}$ is the non-adapted pretrained model.
That is, for each revealed sample/batch $x_{t}$, the model needs to predict its label before receiving the next data point $x_{t+1}$.
The adaptation method $g$ can exploit the predicted label to improve the model performance on the next revealed samples.
The performance of an adaptation method $g$ is measured in an online manner by comparing the predicted label $\hat{y}_{t}$ with the ground truth label $y_{t}$.

## 4 Proposed Solution

Figure: Figure 2: Adapting at test-time with MiDl. At test time, the stream reveals a sample. MiDl uses multimodal samples to adapt by using three copies of the model and one forward pass for each modality combination. MiDl leverages (KL) divergence to align the predictions of the adapted model $f_{\theta_{t}}$ with those of the original model $f_{\theta_{0}}$, ensuring that the adapted model does not deviate too far from the original model’s predictions. The Mutual-Information (MI) component uses the prediction from the different modalities to reduce the dependency on any specific modality, fostering a more generalized and robust prediction across different modality combinations. MiDl updates the model for step $t+1$ using the combination of KL and MI in Equation 2.
Refer to caption: extracted/2404.15161v1/figures/pipeline.png

In this section, we propose our novel adaptation strategy to combat missing modalities at test time.
Recall that the adaptation method $g$ has to satisfy the following requirements.
First, $g$ has to be fully self-supervised. The test stream reveals strictly unlabeled data at test time.
Second, $g$ has to conduct the adaptation in an online manner.
That is, $g$ should adapt on each revealed sample/batch of data $x_{t}$ since $\mathcal{S}$ reveals $x_{t+1}$ only after the model predicts $\hat{y}_{t}$ (refer to the interaction between $\mathcal{S}$ and $g$ in Section 3.1).

To formulate our regularizer, we begin by asking the following question: How should an optimal $f_{\theta}$ behave under missing modality?
We believe that a robust model against missing modality should satisfy two properties.
First, the prediction of $f_{\theta}$ should not depend on the available modality.
Ideally, $f_{\theta}$ should output the same prediction under both complete and incomplete modality, hence satisfying the following equality: (i) $f_{\theta}(x_{i};M=A)=f_{\theta}(x_{i};M=V)=f_{\theta}(x_{i};M=AV)\,\forall i$.
(ii) $f_{\theta}$ should have high accuracy in predicting data with complete modality which is generally satisfied for $f_{\theta_{0}}$.
Satisfying both properties will result in a model that is accurate (satisfying (ii)) and robust against missing modality (satisfying (i)).
To construct an adaptation algorithm that satisfies both properties, we propose to solve the following optimization problem:

$$ $\theta^{*}=\underset{\theta}{\text{arg}\min}\,\underset{{x\sim\mathcal{S}}}{ \mathbb{E}}\,\left[\text{MI}\left(f_{\theta}(x;m),m\right)+\text{KL}\left(f_{ \theta}(x\>|\>M=AV)||f_{\theta_{0}}(x\>|\>M=AV\right)\right]$ (1) $$

where $\text{MI}(u,v)$ is the mutual information between $u$ and $v$ and KL is the KL-Divergence.
Note that if the mutual information between two random variables $\text{MI}(u,v)=0$, then $u$ and $v$ are independent from each other.
That is, minimizing the first term in the objective function in Equation (1) aims to satisfy property (i).
Hence, if $\text{MI}\left(f_{\theta^{*}}(x;m),m\right)=0$, then the output of the adapted classifier becomes independent from the available modality at test time. Furthermore, to ensure that the adapted parameters are still performing well upon adaptation, we equip the mutual information minimization with a self-distillation approach through minimizing KL divergence between the prediction of the adapted model and the original $f_{\theta_{0}}$, satisfying property (ii).

Although the objective function in Equation (1) is self-supervised, obtaining $\theta^{*}$ requires accessing all samples from the stream $\mathcal{S}$ to evaluate the expectation $\mathbb{E}_{x\sim\mathcal{S}}$, which is not available at test time in the online evaluation protocol.
To that end, we approximate the expected value during adaptation at time $t$ with the samples $x_{t}$ revealed from the stream.
Hence, our Mutual information with self-Distillation (MiDl) adaptation step at timestamp $t$ can be expressed as the following gradient descent step:

$$ $\displaystyle\theta_{t+1}=\theta_{t}-\gamma\nabla_{\theta}\mathcal{L}_{\text{ MiDl}}|_{\theta=\theta_{t}}=\theta_{t}-\gamma\nabla_{\theta}(\mathcal{L}_{ \text{MI}}+\mathcal{L}_{\text{KL}})|_{\theta=\theta_{t}}$ (2) $\displaystyle\text{with}\,\,\mathcal{L}_{KL}=\text{KL}\left(f_{\theta}(x_{t};M =AV)||f_{\theta_{0}}(x_{t};M=AV\right)$ $\displaystyle\qquad\mathcal{L}_{\text{MI}}=\underbrace{\mathbb{E}_{m}\left[ \sum_{i=1}^{K}f_{\theta}^{i}(x_{t};m)\log\left(f_{\theta}^{i}(x_{t};m)\right) \right]}_{\mathcal{L}_{\text{ent}}}-\underbrace{\sum_{i=1}^{K}\hat{f}_{\theta} ^{i}(x_{t})\log\left(\hat{f}_{\theta}^{i}(x_{t})\right)}_{\mathcal{L}_{\text{ div}}}$ $$

where $\hat{f}_{\theta}(x_{t})=\mathbb{E}_{m}[f_{\theta}(x_{t};m)]$, $f_{\theta}^{i}(x_{t};m)$ is the $i^{th}$ element in the vector $f_{\theta}(x_{t};m)$, and $\gamma>0$ is the learning rate of the gradient descent step.
To estimate the expectation $\mathbb{E}_{m}[f_{\theta}(x_{t};m)]$, we conduct three forward passes for $x_{t}$ with setting $m\in\{A,V,AV\}$ and average their predictions (refer to Figure 2).
Similarly, We average the entropy of those predictions to calculate $\mathcal{L}_{\text{ent}}$.
Note that $\mathcal{L}_{\text{div}}$ is the entropy of the average prediction across modalities.
Note that under incomplete modality, $\mathcal{L}_{\text{ent}}=\mathcal{L}_{\text{div}}$ resulting in $\mathcal{L}_{\text{MI}}=0$.
Hence, $\mathcal{L}_{\text{MI}}\neq 0$ only when $x_{t}$ has complete modalities (refer to the appendix for further details).
Based on that, we propose to conduct our adaptation step *only* when $\mathcal{S}$ reveals $x_{t}$ with complete modalities.
However, when $\mathcal{S}$ reveals $x_{t}$ with incomplete modality, we refrain from adaptation and set $\theta_{t+1}=\theta_{t}$.
This makes the interaction between the stream $\mathcal{S}$ and our proposed adaptation with MiDl at each time step $t$ take the following form:

For $x_{t}$ with missing modality, we leverage the most recent adapted model to perform predictions without adaptation.
Next, we validate the effectiveness of our proposed MiDl experimentally.

## 5 Experiments

We perform a thorough analysis of MiDl’s performance under different missing modality scenarios. We present our experimental setup in Section 5.1. Then, in Section 5.2, we compare our method on the most simple test-time adaptation scenario where the model has to adapt while replying to the stream. Then, in Section 5.3, we study MiDl’s behaviour under the Long-Term Adaptation (LTA) setup that happens when the model has been exposed to a long stream of data $\mathcal{S}$. Finally, Section 5.4 analyzes the scenario, in which one has access to some unlabeled data from an out-of-domain source before the model is deployed. Under all these three scenarios, we show that MiDl is the better alternative to combating missing modalities at test time.

### 5.1 Setup

We follow [] and use the validation sets of Epic-Kitchens and Epic-Sounds datasets for all experiments and report action recognition accuracy. We also use Ego4D-AR videos for the out-of-domain experiments in Section 5.4.

#### Datasets.

Epic-Kitchens [, ] contains 100 video hours of people performing cooking activities recorded with wearable cameras. The dataset is commonly used for benchmarking audiovisual egocentric action recognition. Each instance is annotated as a noun and verb pair (*e.g*. cut tomato), with a total of 300 noun and 97 verb classes. The validation set contains 9668 samples.
Epic-Sounds [] provides sound-based annotations for the same 100 video hours. It contains 44 classes and 8045 validation set samples.

We assess the effectiveness of the baselines and our proposed MiDl in combating missing modalities at test time.
To do so, we present the pretrained model with a stream $\mathcal{S}$ of unlabeled validation data where we drop one modality with a rate of $1-p_{AV}$.
We set $p_{AV}\in\{0.0,0.25,0.5,0.75,1.0\}$ resulting in a missing rate of $\{100\%,75\%,50\%,25\%,0\%\}$, respectively. Following [], for each dataset, we drop the primary modality (*i.e.* sound for Epic-Sounds and video for Epic-Kitchens). Thus, for a 75% missing rate (*i.e.* $p_{AV}=0.25$), the stream containing Epic-Sounds has $P=\{0.0,0.75,0.25\}$ and $P=\{0.75,0.0,0.25\}$ for Epic-Kitchens.

#### Architecture.

Unless stated otherwise, we use the stronger architecture Multimodal Bottleneck Transformer (MBT) [] as $f_{\theta}$. We use the baselines of [] trained on Epic-Sounds and Epic-Kitchens.
Nonetheless, we experiment with the multimodal fusion via vanilla self-attention architecture [] in Section 6.1.

### 5.2 MiDl Improves Performance

**Table 1: Combating missing modalities at Test-time. We showcase different results of different methods when combating missing modalities at test time. The first two rows show the unimodal performance and the Multimodal Baseline with no adaptation. Adapting to missing modalities is critical since the performance of the Baselines degrades quickly under increasing missing modalities. We show three alternative test-time adaptation methods and demonstrate that our proposed MiDl is effective at combating missing modalities at test time, outperforming every other alternative.**
|  | Epic-Sounds (%) | Epic-Kitchens (%) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model<br>$1-p_{AV}$ | 0 | 25 | 50 | 75 | 100 | 0 | 25 | 50 | 75 | 100 |
| Unimodal | 41.4 | 40.0 |  |  |  |  |  |  |  |  |
| Baseline | 55.1 | 45.6 | 37.1 | 28.3 | 19.5 | 63.9 | 55.5 | 46.8 | 37.9 | 29.5 |
| +Shot | 55.0 | 45.6 | 37.1 | 28.7 | 20.1 | 63.8 | 56.0 | 48.5 | 41.1 | 34.5 |
| +Tent | 54.9 | 45.0 | 35.9 | 26.3 | 17.7 | 63.8 | 53.7 | 38.6 | 23.6 | 9.8 |
| +ETA | 55.2 | 45.6 | 37.0 | 28.3 | 19.5 | 63.4 | 52.2 | 34.1 | 19.5 | 7.9 |
| +MiDl (ours) | 55.1 | 46.7 | 38.9 | 29.9 | 19.5 | 63.8 | 58.4 | 52.1 | 44.9 | 29.5 |

Table 1 reports the average accuracy under different missing modality rates on Epic-Sounds and Epic-Kitchens.
In this experiment, we compare our proposed MiDl against three off-the-shelf test-time adaptation methods.
Namely, we include the information maximization approach, Shot [], and the entropy minimization with data selection from ETA [] (refer to the appendix for their implementation details).
For a fair comparison, we assess the efficacy of the aforementioned adaptation methods against MiDl (refer to Section 4).

We observe that: (i) our proposed MiDl significantly enhances the performance of pretrained models under missing modality.
We record a significant gain of 5% and 7% on the Epic-Kitchens dataset under missing rates of 50% and 75%, respectively.
On the more challenging Epic-Sounds, MiDl boosts the accuracy of the baseline from 37.1% and 28.3% to 38.9% and 29.9% under the same missing rates, respectively.
Note that this performance boost comes at no cost of retraining the model; simply adapting it during test time.
This result demonstrates how our proposed mutual information minimization encourages the model to become invariant to domain shifts due to missing modality.
(ii) MiDl recovers the performance of the baseline under complete modality (retaining the 55.1% and 63.8% accuracy under 0% missing rate on Epic-Sounds and Epic-Kitchens, respectively).
That is, our proposed KL divergence regularization enhances the information retention of the original pretrained model under complete modality inference.
(iii) We also observe that off-the-shelf test-time adaptation methods are ineffective in this setup.
This limitation arises because all TTA methods are designed to tackle covariate domain shifts and, thus, are not tailored to enhance performance under this specific type of domain shift (missing modality).
It is noteworthy to mention that MiDl does not improve nor degrade the performance under 100% missing rate (*i.e.* $p_{AV}=1$).
This is because our adaptation method only update the model parameters under the presence of data with complete modality.

### 5.3 Performance Under Long-Term Adaptation (LTA)

**Table 2: Adaptation at Test-time under Long-term Adaptation. We showcase the results of MiDL under the assumption that the stream of data is very long. We use unlabeled data to simulate a longer stream and report results on the validation set of each dataset. Our MiDl benefits from long-term adaptation. Especially at higher missing rates (>75%).**
|  | Epic Sounds(%) | Epic Kitchen(%) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model<br>$1-p_{AV}$ | 0 | 25 | 50 | 75 | 100 | 0 | 25 | 50 | 75 | 100 |
| Unimodal | 41.4 | 40.0 |  |  |  |  |  |  |  |  |
| Baseline | 55.1 | 45.6 | 37.1 | 28.3 | 19.5 | 63.9 | 55.5 | 46.8 | 37.9 | 29.5 |
| +MiDl (ours) | 55.1 | 46.7 | 38.9 | 29.9 | 19.5 | 63.8 | 58.4 | 52.1 | 44.9 | 29.5 |
| + MiDl - LTA (ours) | 55.1 | 46.9 | 39.5 | 32.6 | 26.2 | 63.8 | 58.4 | 52.4 | 46.7 | 41.4 |

Next, we analyze the effectiveness of our proposed MiDl under a long stream of data $\mathcal{S}$.
Since MiDl operates at test time, its performance gain can vary depending on the data revealed at test time.
We suspect that if $\mathcal{S}$ reveals more *unlabeled* data with complete modality at test time, MiDl will adapt the model parameters for more iterations enhancing its invariance properties against missing modality.

To study this interesting setting, we allow MiDl to access unlabeled data with complete modality in the form of a stream, denoted as $\mathcal{S}_{\text{in}}$, to perform adaptation.
This simulates the case where the test stream $\mathcal{S}$ is very long where the portion of data with missing modality is still big enough.
We then ask the following question: how would MiDl perform after this long adaptation?
We let $\mathcal{S}_{\text{in}}$ be a subset of training data, and test the model on $\mathcal{S}$ being the validation set, following the standard evaluation in Section 5.2.

Table 2 summarizes the results on Epic-Sounds and Epic-Kitchens datasets at the same missing rates considered in Section 5.2.
We observe: (iv) the longer the stream is, MiDl provides a bigger performance gain.
For example, MiDl further improves the non-adapted baseline by 4.3% and 8.8% on Epic-Sounds and Epic-Kitchens, respectively, under a missing rate of 75% (*i.e.* $p_{AV}=0.25$).
In addition, even under 100% missing rate, MiDl improves the accuracy by a notable margin of 6.7% and 11.9% on Epic-Sounds and Epic-Kitchens, respectively.
That is, the adaptation on $\mathcal{S}_{\text{in}}$ unlocks a bigger potential of MiDl for life-long adaptation even when $\mathcal{S}$ reveals data with a single modality.
(v) Unlike MiDl, other adaptation approaches do not benefit from this long stream setup as their objective functions do not promote building an invariant model against missing modality. We present these results in the supplementary material.

### 5.4 Warm-up on Ego4D: Exploiting Out of Domain Adaptation

**Table 3: Adaptation at Test-time with Ego4D warm-up. We show another use cause of MiDL, in which the assumption is having access to out-of-domain unlabeled data to adapt before deployment. The results showcase MiDL’s capabilities on leveraging unlabeled-out-of-domain data to combat missing modalities.**
|  | Epic-Sounds (%) | Epic-Kitchens (%) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model<br>$1-p_{AV}$ | 0 | 25 | 50 | 75 | 100 | 0 | 25 | 50 | 75 | 100 |
| Baseline | 55.1 | 45.6 | 37.1 | 28.3 | 19.5 | 63.9 | 55.5 | 46.8 | 37.9 | 29.5 |
| + MiDl (ours) | 55.1 | 46.7 | 38.9 | 29.9 | 19.5 | 63.8 | 58.4 | 52.1 | 44.9 | 29.5 |
| + Ego4D Warm-up | 55.0 | 46.6 | 38.6 | 30.9 | 19.5 | 63.7 | 58.4 | 52.4 | 46.7 | 37.8 |

Next, we analyze another practical setup.
In particular, we consider the case when not only the pretrained model $f_{\theta}$ is provided, but unlabeled data from out-of-domain, denoted as $\mathcal{S}_{\text{out}}$ can also be accessible.
Note that our optimization objective in Equation (1) does not assume the data distribution we adapt to.
This then begs the question:
would adapting with MiDl on a different data distribution $\mathcal{S}_{\text{out}}$ help combat missing modalities in $\mathcal{S}$?

To answer this question, we leverage the recent Ego4D [] data. Although Ego4D has egocentric videos, they come from very different environments and scenarios that deviate from the usual kitchen scene. These changes introduce additional domain shifts when evaluating on Epic-Sounds and Epic-Kitchens.
We set $\mathcal{S}_{\text{out}}$ to be 25% of the Ego4D-AR and present them to the model in the form of a stream. It is worth noting that, in this case, we keep using our self-supervised MiDL objective, and we do not require any labels from Ego4D-AR.

We replicate our setup from Section 5.3 and perform adaptation on $\mathcal{S}_{\text{out}}$ followed by evaluation on $\mathcal{S}$ (*i.e.* evaluation on the standard validation set of Epic-Kitchens and Epic-Sounds).
We name the adaptation phase on $\mathcal{S}_{\text{out}}$ as a warm-up phase.

Table 3 summarizes the results where we show the performance of the non-adapted baseline, our proposed MiDl when adapted solely on $\mathcal{S}$, and our MiDL equipped with adaptation on $\mathcal{S}_{\text{out}}$.
We observe that (vi) conducting a warm-up generally exerts a positive influence on overall performance in cases of missing modality.
The enhanced version of MiDl with Ego4D warm-up improves over MiDl by 1% and 1.8% on Epic-Sounds and Epic-Kitchens, respectively, under a missing rate of 75%.
Furthermore, we observe that even under 100% missing rate, adapting on $\mathcal{S}_{\text{out}}$ enhances the accuracy of MiDl by an impressive 8% on Epic-Kitchens.
This demonstrates the versatility of MiDl, which provides consistent performance gains under different setups.

#### Takeaway.

In this section, we showcased the effectiveness of our proposed MiDl in combating the missing modality challenge at test time (Section 5.2).
Further, we analyzed the impact of long-term adaptation where MiDl provided further performance gain where the relative improvement is up to 30% (Section 5.3).
At last, we showed how adapting with MiDl on out-of-distribution data, mimicking data scarcity situations, still boosts the model’s accuracy (Section 5.4).

## 6 Analysis on MiDl

In this section, we conduct a comprehensive analysis of MiDl.
In particular, we show how our proposed test-time adaptation is agnostic to the choice of $f_{\theta}$ (Section 6.1), the missing modality (Section 6.2). We conclude by analyzing the different components of MiDl in Section 6.3 and its computational requirements in Section 6.4.

### 6.1 Agnostic to Architecture Choice

**Table 4: MiDL performance under a different architecture. We showcase the effectiveness of MiDL when applied to a distinct baseline architecture characterized by multimodal self-attention. MiDL enhances performance across all missing rates, underscoring its robustness and adaptability to various underlying architectures.**
| Model<br>$1-p_{AV}$ | Epic-Sounds (%) |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 0 | 25 | 50 | 75 | 100 |  |
| Self-Attention Baseline | 45.3 | 38.8 | 32.7 | 26.7 | 20.5 |
| +Shot | 45.5 | 39.0 | 32.8 | 26.8 | 20.7 |
| +Tent | 45.3 | 38.6 | 32.3 | 26.0 | 19.8 |
| +ETA | 45.3 | 38.8 | 32.7 | 26.7 | 20.5 |
| +MiDl (ours) | 45.5 | 39.5 | 33.8 | 27.5 | 20.5 |
| +MiDl - LTA (ours) | 45.5 | 39.6 | 34.5 | 29.0 | 23.2 |

Here, we analyze the robustness of MiDl against the architecture choice $f_{\theta}$.
To this end, we replicate our experimental setup from Sections 5.2 and 5.3 but set $f_{\theta}$ to be the multimodal fusion via vanilla self-attention architecture [], as opposed to the MBT architecture [].

Table 4 summarizes the results under different missing rates.
We compare the non-adapted baseline, Shot and ETA, and our proposed MiDl.
We observe that MiDl provides a consistent performance boost under the self-attention architecture, similar to our observations with the MBT architecture in Section 5.2.
For example, MiDl improves the accuracy of the baseline under 50% missing rate by a notable 1.1% without affecting the performance under complete modality.
Further, we analyze the performance of MiDl under Long-Term Adaptation (LTA), following the setup in Section 5.3.
Similar to our earlier findings, LTA unlocks a better potential of MiDl where further performance gain is attained.
Under this setup, MiDl improves the baseline by 2% under 50% and 75% missing rates, and by 2.7% under 100% missing rates.
These results show that MiDl is agnostic to the choice of architecture.

### 6.2 Agnostic to the Type of Missing Modality

**Table 5: Adaptation at Test-time - Other Missing Modalities. In this table we show the results using the complementary modality for each of the datset, *i.e*. video for Epic Sounds and Audio for Epic Kitchens. We observe that MiDL improves consistently under this setup, highlighting its robustness to different types of modalities missing at test time.**
|  | Epic Sounds (%) | Epic Kitchens (%) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model<br>$1-p_{AV}$ | 0 | 25 | 50 | 75 | 100 | 0 | 25 | 50 | 75 | 100 |
| Unimodal | 46.5 | 63.2 |  |  |  |  |  |  |  |  |
| Baseline | 55.1 | 53.4 | 51.8 | 50.5 | 48.8 | 63.9 | 61.0 | 58.0 | 54.8 | 52.1 |
| +MiDl | 55.1 | 53.3 | 51.8 | 50.7 | 48.8 | 63.9 | 61.6 | 59.1 | 55.9 | 52.1 |
| +MiDl - LTA | 55.1 | 53.4 | 52.0 | 51.0 | 49.4 | 63.9 | 61.7 | 59.5 | 57.3 | 55.3 |

In previous sections, we focused our experiments on the effectiveness of MiDl when the dominant modality is missing.
In particular, we analyzed dropping the audio modality for Epic-Sounds and the visual modality for Epic-Kitchens.
In this section, we attempt to study the robustness of MiDl against dropping the other modality at test time.
To this end, we conduct experiments on Epic-Sounds, where we drop the visual modality, and on Epic-Kitchens, where we drop the audio modality.
Note that, under this setup, the baseline experiences less performance degradation, similar to scenarios involving the missing dominant modality.
We replicate our experimental setup from Section 6.1 where we compare the non-adapted baseline, its equipped version with our proposed MiDl, and the LTA effect of MiDl.

Table 5 summarizes the results.
Our experiments show that MiDl consistently enhances the performance of the pretrained model irrespective of the type of missing modality.
For example, MiDl improves the baseline by 1% on Epic Kitchens under missing rates of 50% and 75%.
Further, and similar to our observations in Section 5.3 and 6.1, the LTA further improves MiDl by over 3% under 100% missing rate (*i.e.* $p_{AV}=0$).

### 6.3 Components of MiDl

**Table 6: Analyzing MiDl components. We analyze the different components of MiDL. When the Mutual-Information (MI) component is missing, the model does not have any reason to adapt since the KL divergence is maximized by predicting the same as the base model. When KL is not present, the MI alone deviates from the initial results and performs poorly under higher missing rates.**
|  |  |  | Epic Sounds (%) | Epic Kitchens (%) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model<br>$1-p_{AV}$ | $\mathcal{L}_{\text{MI}}$ | $\mathcal{L}_{\text{KL}}$ | 0 | 25 | 50 | 75 | 0 | 25 | 50 | 75 |
| Baseline | ✗ | ✗ | 55.2 | 45.6 | 37.1 | 28.3 | 63.9 | 55.5 | 46.8 | 37.9 |
| + Dl | ✗ | ✓ | 55.2 | 45.6 | 37.1 | 28.3 | 63.9 | 55.5 | 46.8 | 37.9 |
| + Mi | ✓ | ✗ | 40.4 | 39.3 | 36.1 | 29.6 | 53.5 | 50.5 | 47.6 | 45.9 |
| +MiDl (ours) | ✓ | ✓ | 55.1 | 46.7 | 38.9 | 29.9 | 63.8 | 58.4 | 52.1 | 44.9 |

At last, we ablate the effectiveness of each component of MiDl.
Recall from our adaptation step in Equation (2) that MiDl has two components: (1) mutual information minimization and self-distillation through minimizing KL-divergence.
We believe that the success of MiDl is attributed to the interplay between both components.

To analyze the importance of each component, we adapt the pretrained baseline with each loss function independently and compare the performance under different missing rates.
Table 6 summarizes the results on Epic-Sounds and Epic-Kitchens datasets with $p_{AV}\in\{0.25,0.5,0.75,1.0\}$.
We observe that adapting solely with self-distillation (*i.e.* minimizing $\mathcal{L}_{\text{KL}}$) results in no adaptation.
On the contrary, adaptation by minimizing only $\mathcal{L}_{\text{MI}}$ results in a significant performance drop.
While minimizing $\mathcal{L}_{\text{MI}}$ can indeed result in $f_{\theta}$ that is robust against missing modality, it is not necessarily accurate (*i.e.* satisfying the first (i) property in 4 and violating the second (ii)).
MiDl balances information minimization with the information retention loss $\mathcal{L}_{\text{KL}}$.

### 6.4 Computational Requirement of MiDl

While our previous results have demonstrated the efficacy of MiDl in providing performance gains, we note that this improvement comes at a computational cost.
In fact, conducting an adaptation step with MiDl requires conducting a total of 3 inferences through $f_{\theta_{t}}$ and an inference through the initial pretrained model $f_{\theta_{0}}$, followed by a single backward pass.
This makes an inference through MiDl 5$\times$ more expensive than doing inference without any adaptation.
In practice, the latency of MiDl is only 2$\times$ slower than the non-adapted model since all the additional 4 forward passes can be performed in parallel.

#### Takeaway.

In this section, we conducted comprehensive analysis on our proposed MiDl.
We showed that MiDl is agnostic to the choice of architecture (Section 6.1) and the type of missing modality (Section 6.2).
We further analyzed the importance of both components of MiDl in Section 6.3 and its computational requirements in Section 6.4.

## 7 Conclusions

In this work, we presented MiDl, a new method for improving how pretrained video recognition models handle missing modalities at test time. MiDl improves the model’s ability to give accurate predictions regardless of the availability of modalities by minimizing mutual information and using self-distillation. Our experiments show that MiDl can significantly increase accuracy across various datasets and scenarios and under various missing rates, making it a practical solution for real-world applications dealing with incomplete modalities.

## Contribution Statement

We highlight the contribution of each author in this project.
Merey Ramazanova coordinated and refined the project’s focus on missing modality issues in egocentric videos. She led the implementation and experimental design, including MiDl and other baselines, managed data preparation, and co-designed the evaluation protocols. She proposed the Out of Domain Adaptation setup using Ego4D.
Alejandro Pardo refined the code for large-scale experimentation, assisted in the baseline implementation, conducted the experiments, and explored various ablations and parameters for each method. He played a significant role in writing and presenting the experimental section of the work.
Bernard Ghanem guided the project from the beginning, helped shape the initial ideas, and significantly contributed to the design of the experiments and the project’s overall development. He also proofread and polished the manuscript.
Motasem Alfarra initiated the project by formulating the missing modality challenge as a test-time adaptation problem, proposed the MiDl objective function, co-designed the evaluation protocols, led the writing efforts, and advised the implementation of MiDl along with the other baselines.

## Appendix 0.A Proposed Solution: Extended Discussion

In Section 4, we proposed our novel adaptation strategy; MiDl.
First, note that our intuition of minimizing the mutual information between the output of the network and the modality source comes from the following observation:
Let $X$ and $Y$ be two random variables, then

$$ If $\displaystyle\text{MI}(X,Y)=0\,\,\rightarrow\,\,X\text{ and }Y\text{ are independent}.$ Proof: $\displaystyle\text{MI}(X,Y)=\sum_{x\in X}\sum_{y\in Y}P_{XY}(x,y)\log\left( \frac{P_{XY}(x,y)}{P_{X}(x)P_{Y}(y)}\right)$ If $\displaystyle\text{MI}(X,Y)=0\,\,\rightarrow\,\,P_{XY}(x,y)=P_{X}(x)P_{Y}(y)$ $$

That is, minimizing the mutual information between the output prediction of $f_{\theta}$ and the available modality should make the adapted network robust against missing modality.
Second, MiDl adapts the pretrained model $f_{\theta}$ on the unlabeled data revealed from the stream only if the revealed data is with complete modality.
This is since: (1) the KL-divergence loss only applies on data with full modality and (2) the information minimization loss only operates under complete modality.
To wit, let $x_{t}$ be revealed with $m=A$.
Then the estimate of our $\mathcal{L}_{\text{MI}}$ will be

$$ $\displaystyle\mathcal{L}_{\text{MI}}=$ $\displaystyle\underbrace{\sum_{i=1}^{K}f_{\theta}^{i}(x_{t};M=A)\log\left(f_{ \theta}^{i}(x_{t};M=A)\right)}_{\mathcal{L}_{\text{ent}}}$ $\displaystyle-$ $\displaystyle\underbrace{\sum_{i=1}^{K}f_{\theta}^{i}(x_{t};M=A)\log\left(f_{ \theta}^{i}(x_{t};M=A)\right)}_{\mathcal{L}_{\text{div}}}=0.$ $$

This is since $\mathbb{E}_{m}$ will be estimated with a single point at $m=A$.
Thus, MiDl adapts the parameters only when $\mathcal{S}$ reveals data with complete modality at test time.

## Appendix 0.B Experiments

### 0.B.1 Implementation Details

In Section 5.1, we detailed our experimental setup.
Here, and for reproducibility, we provide the implementation details for MiDl as well as the baselines.
Note that for all considered adaptation methods, we follow the standard practice in the test-time adaptation literature [, , , ] and only adapt the learnable parameters of the normalization layers.
We freeze the rest of the model parameters during the update step.
Further, for MiDl, we balance the mutual information loss and the self-distillation loss through:

$$ $\mathcal{L}_{\text{MiDl}}=\lambda_{1}\mathcal{L}_{\text{MI}}+\lambda_{2} \mathcal{L}_{\text{KL}}.$ $$

Note that for all our experiments, we set $\lambda_{1}=\lambda_{2}=3$.
Further, we conduct the adaptation step with an SGD [] step with a learning rate of $25\times 10^{-4}$, and a momentum of $0.9$, following [, , ].

Regarding the considered test-time adaptation baselines, we considered the entropy minimization approach known as Tent [], its improved version equipped with data-selection process ETA [], and the information maximization Shot [].
We followed the official implementation of each method and used the recommended hyperparamaters.

### 0.B.2 Baselines Under Long Term Adaptation

**Table 7: Adaptation at Test-time under Long-term Adaptation. We showcase the results of MiDL under the assumption that the data stream is very long. We use unlabeled data to simulate a longer stream and report results on the validation set of each dataset. Our MiDl benefits from long-term adaptation. Especially at higher missing rates (>75%).**
|  | Epic-Sounds (%) | Epic-Kitchens (%) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model<br>$1-p_{AV}$ | 0 | 25 | 50 | 75 | 100 | 0 | 25 | 50 | 75 | 100 |
| Unimodal | 41.4 | 40.0 |  |  |  |  |  |  |  |  |
| Baseline | 55.1 | 45.6 | 37.1 | 28.3 | 19.5 | 63.9 | 55.5 | 46.8 | 37.9 | 29.5 |
| +Shot - LTA | 55.0 | 45.6 | 37.2 | 28.7 | 20.3 | 63.8 | 56.0 | 48.2 | 41.0 | 34.4 |
| +Tent - LTA | 54.5 | 44.6 | 35.5 | 26.1 | 17.7 | 62.7 | 53.5 | 40.0 | 25.8 | 12.3 |
| +ETA - LTA | 55.0 | 45.5 | 37.0 | 28.2 | 19.5 | 60.9 | 48.8 | 33.0 | 19.3 | 7.5 |
| +MiDl - LTA (ours) | 55.1 | 46.9 | 39.5 | 32.6 | 26.2 | 63.8 | 58.4 | 52.4 | 46.7 | 41.4 |

**Table 8: Adaptation at Test-time - Updating all parameters. In this table, we show the results when we unfreeze all network parameters, not only the normalization layer. We observe that there is no significant difference when compared to updating only the normalization layers.**
|  | Epic-Kitchens (%) |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Model<br>$1-p_{AV}$ | 0 | 25 | 50 | 75 | 100 |
| Unimodal | 40.0 |  |  |  |  |
| Baseline | 63.9 | 55.5 | 46.8 | 37.9 | 29.5 |
| +MiDl (all parameters) | 63.6 | 58.4 | 52.4 | 46.3 | 29.5 |
| +MiDl (norm layers) | 63.8 | 58.4 | 52.1 | 44.9 | 29.5 |
| +MiDl - LTA (all parameters) | 63.6 | 58.3 | 52.5 | 47.1 | 42.0 |
| +MiDl - LTA (norm layers) | 63.8 | 58.4 | 52.4 | 46.7 | 41.4 |

We extend our comparison against the considered baselines under the Long Term Adaptation (LTA) setup.
We replicate our experimental setup in Section 5.3 and compare MiDl against SHOT, Tent, and ETA.
We report the results in Table 7 for the MBT architecture.
We observe consistent findings in Section 5.3, where naive adaptation baselines do not benefit from this long-term adaptation.
Further, we find that MiDl is the only adaptation method that provides further performance gains under this LTA setup.

### 0.B.3 Adapting More Layers

At last, we extend our analysis on MiDl to include the effect of adapting more parameters.
In particular, we compare adapting only the learnable parameters of the normalization layers against adapting the whole network parameters.
We report the result in Table 8 on Epic-Kitchens.
We observe that adapting all network parameters with MiDl results in a minor performance gain.
For instance, under the 75% missing rate, adapting all parameters improves over adapting only the normalization layers by 0.4% under the LTA setup and by 1.4% with the test-time adaptation of MiDl.
We note here that this comes at a computational expense as it is faster and more efficient to adapt only the normalization layers.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. If this approach faces extreme examples, such as a video showing a calm street while the audio is an explosion, will this model mislead the baseline model into the wrong direction?
2. You might consider adding extra blocks to the model, so that if updates are needed, only the added portions need to be updated. Alternatively, updating part of the model's structure could prevent the significant latency introduced by updating the entire system.
soundness: 3
strengths: The paper redefines the modality missing problem as a test-time adaptation (TTA) issue, emphasizing the challenges of modality absence faced in online multimodal tasks. This is indeed an urgent problem that needs to be addressed for many online multimodal tasks.

The proposed MiDl method effectively enhances the baseline model's ability to handle missing modalities, serving as a solution for the modality missing problem in multimodal online tasks. This approach can act as a supplement when facing modality absence in such tasks. For instance, if modalities are functioning normally, this pipeline may not be used; however, when a modality is missing, the proposed solution can improve the baseline model's capability to handle the missing modalities. Additionally, normal inputs and prediction results can serve as supplementary information when modalities are insufficient.

The experiments presented in the paper are comprehensive, demonstrating that the 
method is independent of modality selection, baseline models, and model frameworks, thereby proving the robustness of the proposed solution.
summary: To tackle the issue of modality missing in real-time tasks, this framework offers an online self-supervised learning method called MiDl. MiDl uses mutual information and KL divergence as loss functions to optimize the model in real time, enabling the baseline model to better handle inputs with missing modalities.
weaknesses: 1. "First, the prediction of should be invariant to the modality source . Ideally, f_{\theta} should output the same prediction under both complete and incomplete modality, hence satisfying the following equality: (i)...," The underlying assumption of the approach is controversial. The task will degenerate into a modality distillation problem if this assumption holds,. Is there a more reasonable way to phrase this?
2. Implementing this method in real-world production could introduce significant computational overhead and latency. Normal models can be accelerated through techniques like compression and distillation, but this approach involves updating model weights, requiring the retention of the complete model, making it difficult to deploy directly in practice.
3. Could you include experiments demonstrating the approach's decision-making in more complex online scenarios? The experiments provided in the paper do not represent the best use case for this method; its most suitable application is in online scenarios, so experiments in these contexts would better support the results.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the Weakenesses section. I highly encourage the authors to directly include the code for the verification of reproducibility.
soundness: 3
strengths: Overall, the paper is interesting and easy to follow. 
- The formulation of the test-time adaptation for tackling missing modality without the need for model retraining is indeed novel and can be foreseen to be applied to various applications which contains multi-modal information.
- Although the method itself is not complex and consists of components already used for various tasks for multi-modal learning and egocentric video analysis, they are leveraged in MiDl with strong motivation which are intuitive and reasonable. The extended discussion also offers a deeper understanding over the formulation of MiDl. This method could prove to be a good starting point for subsequent discussions in the research community.
- The authors also provide a comprehensive analysis over the performance of MiDl in the formulated task, and also benchmarked previous methods such as SHOT, TENT under the same setting, which also provides a further insight into the challenges and possible methods to further tackle the task.

In general, this paper is relatively well presented with a simple yet highly motivated method for an interesting formulation of a realistic challenge.
summary: This paper tackles on missing modalities in egocentric videos without the need to retrain models by formulating this challenge as a test-time adaptation task. The authors proposed MiDl which minimizes the mutual information between prediction and the modality, with the incorporation of self-distillation to maintain performance when all modalities are available. The author benchmarked several methods under such problem formulation, demonstrating a descent performance when part of the modality are missing in two egocentric video datasets.
weaknesses: There are a few minor concerns remaining in the paper, mainly on the clarity and possible extension in discussion of the proposed method. I would like the authors to consider the following concerns if possible:
1. On Page 4, Line 197-198, the author states that "$f_\theta$ should retain high performance in predicting data with complete modality, which is generally satisfied for $f_{\theta_0}$". Does this imply that the non-adapted pretrained model must be pretrained with all modalities available? What if the pretrained model is only trained with a single modality (e.g., only with visual information without the audio information which is rather common in video models)?
2. It is observed that there is a large drop when $1-p_{AV}=100$, where none of the data contain both modalities. What would be a possible approach to mitigate this drop in performance. It is observed that the drop for MiDl is significantly more severe than that of SHOT.
3. The current method only touches upon the case for two modalities (audio and video), is it expandable towards more modalities. Also, are there limitations for the possible types of modalities or it can be any modalities as long as they are obtained from the same set of data?
4. The experiments are performed for each dataset with a drop in the primary modality, what would be the result if the secondary modality is dropped with the same probability?
5. Lastly, the code is currently NOT available, which means that the reproducibility of the result is not verified.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 4
questions: While the technical aspects and experimental results are generally strong, there are areas for improvement in the motivation, clarity of presentation, and some experimental details. 

I presented many questions and suggestions in the weaknesses suggestions. In particular, I would suggest the authors focus on the concerns about the motivation and the experiments aligning with that motivation. My comments regarding notation and small fixes are merely suggestions.
soundness: 3
strengths: Introduction:
1. The second and third paragraphs effectively identify the gap in the literature and provide a robust overview of the proposed solution.

Related Works:

2. Related works is concise and relevant

Missing Modality as Test Time Adaptation:

3. This section is well-written and easily comprehensible.

Experiments:

4. It is commendable that the experiments were repeated 5 times with reported standard deviations in Table 11; the results appear experimentally robust and convincing.
5. The Ego4d Warmup result on 100% missing rate in the original distribution is an interesting finding with potentially strong applications.

Analysis on MiDL:

6. The Components of MiDl section is strong, and the ablations are empirically sound.
summary: This paper presents a novel approach to handling missing modalities in multimodal learning using test-time adaptation. The method, MiDl, shows promising results across the EPIC kitchen and EPIC sounds datasets, and the method is motivated by the theoretical intuition of minimizing the mutual information between the predicted and available modality. The authors also provide some interesting analysis of the model through long-term adaptation, out-of-distribution warmup, and various ablation experiments. 

This review follows the sections of the paper.
weaknesses: Introduction:
1. The motivation presented in the first paragraph is weak. For instance, in Line 40, could you provide an example or application where inference must occur on a redacted modality? I can think of the application of blurring faces in images or deleting private details in medical records, but it's unclear when only one modality would be completely removed for privacy reasons while the other remains intact for the same data instance. Additionally, the relevance of using cheaper modalities to missing modalities is not apparent (line 40). It would be particularly convincing if the motivation aligned with the tested dataset. For example, if using Epic Kitchens, perhaps a scenario involving a humanoid with a malfunctioning audio sensor, or smart glasses with an obscured camera due to steam or food spillage could be considered.
2. Contribution (3) appears to be describing MiDl, which is already covered in contribution (2). I would recommend reassessing what could constitute a third distinct contribution from your work.
3.Figure 1 requires improvement. The concept of a "potential performance trajectory" needs clarification - is this your hypothesis? This graphic would be more persuasive if it depicted your actual no-adaptation baseline and your TTA method. The purpose of the black line in the middle of the graph is unclear.

Proposed Solution:

4. The notation in eq (1) lacks precision. It is not evident that f(x;m) and m are random variables. The output of f is a distribution. Are you considering this as a random variable, with the value being the indices and the probability of those values the logits? Consider introducing a random variable Y ~ f(x;m). Also, consider using capital letter notation (e.g. "M") for the random variables. Furthermore, how can you evaluate the KL if x ~ S only has the modality m, not AV? Later in this section, it becomes apparent that you only update the model on complete instances. This limitation/assumption should be made clearer in the introduction or Takeaways subsection. This method would only be applicable for testing data that includes some multimodal instances.
5. At last line of page 4, is $x_t$ a single sample? Do you mean samples $x_0 \dots x_t$?

Experiments:

6. Additional details could enhance the reproducibility of this work. Was any hyperparameter tuning conducted for MiDl? Section B.1 mentions the recommended hyperparameters for the baseline but doesn't specify how they were determined for MiDl. Moreover, what were the proportions of the train/val/test split?
7. In section 5.3 LTA: You allow the model to retrain on some of the unlabeled training data. Why not gather $S_{in}$ from the validation set? In this setting, is MiDl trained on $D \ S_{in}$? Or is $S_{in}$ still included in the labeled dataset before test time and then used without labels during test time?
8. Can this method be applied to instances where either modality is missing, e.g., P = {.3,.3,.3}? It would be great to see results for experiment with such a ratio. Currently, it may be the case that the model learns to leverage ONLY the modality that is consistently present in both the complete modality test case and the missing modality test case. In this scenario, would a given unimodal model for the always-present modality perform optimally? Table 1 could be improved by clarifying what is meant by "Unimodal" and why it is only present at the 50% missing rate. For Epic Sounds, is the unimodal the always-present modality (video)?

Analysis on MiDL:

9. Both architecture choices are transformer-based. It would have been more convincing to see a greater diversity of architectures (such as a convolution backbone). Instead of presenting different missing rates as columns in Table 3, it would have been preferable to see different architectures/methods as the columns with a fixed missing rate (perhaps 50%).
10. Given that the main motivation was to avoid retraining an existing method on a large dataset to perform missing modality adaptation, the results would have been more convincing if the authors had either used an existing model+dataset and just performed adaptation, as opposed to training from scratch an existing method. Alternatively, they could have tested with a very large dataset that was computationally expensive. The omnivore pretraining test is good. Did you train the model from scratch on your dataset or use an existing model and apply MiDl?
11. In Table 6, shouldn't 55.2 in the Dl column be bolded?
12. I thought the motivation was that retraining on the train set is computationally expensive, and TTA will prevent that? It's good that you acknowledge the computational requirements of MiDl, but then in the abstract, you shouldn't state: "Current methods, while effective, often necessitate retraining the model... making them computationally intensive." Alternatively, compare your inference computation here with the amount of computation required to retrain training data (to get sufficient performance).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could the authors include comparisons with other approaches specifically addressing the missing modality issue, such as those proposed by Dai et al. (2024), Lee et al. (2019), and Wang et al. (2023)?

2. Given the importance of this task, would the authors consider expanding the benchmark by including more test-time adaptation (TTA) approaches, such as Chen et al. (2022), Wang et al. (2020), Yuan et al. (2023), and Niu et al. (2022), and analyzing performance across different clusters of approaches? Including missing modality approaches as baselines may also strengthen the benchmark.

3. Could the authors provide qualitative results comparing their approach to the baseline, along with a failure case analysis to offer insights into scenarios where the method may fall short?

4. Has the generalizability of the proposed approach been tested on datasets beyond EPIC Kitchen? If not, would the authors consider verifying the performance on additional datasets?

5. Could the authors use TSNE visualization on the latent space to illustrate how the proposed supervision affects feature learning? Specifically, visualizing changes over different epochs in comparison to the baseline might provide additional insights.
soundness: 3
strengths: 1. Missing modality issue is important for test time adaptation of ego centric action recognition. This task will contribute to the community.

2. Method section is clearly written and easy to follow.

3. Compared with the baseline, the proposed approach show good performance on this new task.
summary: In this work, the authors focus on an important task which is test time adaptation for egocentric video action recognition under missing modality. The authors validate existing work of TTA on this new task and propose a new method MiD1 to enhance the robustness of the learned features. The performance of the proposed method is evaluated on the EpicKitchen sound and video dataset
weaknesses: 1. Lack of the comparison with other approaches specifically targeted at missing modality issue. 

a. Dai, Y., Chen, H., Du, J., Wang, R., Chen, S., Wang, H., & Lee, C. H. (2024). A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 27445-27455).

b. Lee, H. C., Lin, C. Y., Hsu, P. C., & Hsu, W. H. (2019, May). Audio feature generation for missing modality problem in video action recognition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 3956-3960). IEEE.

c. Wang, H., Chen, Y., Ma, C., Avery, J., Hull, L., & Carneiro, G. (2023). Multi-modal learning with missing modality via shared-specific feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15878-15887).


2. The authors are suggested to enlarge the benchmarks. I aggree that this task is an important task, however the experiments are limited in this paper which will be harmful to its soundness. The authors could enrich the benchmark using more existing TTA approaches, e.g., d,e,f, and g, and try to provide an analysis on the performance on different cluster of approaches. Missing modality works can also serve as good baselines to enrich the benchmark.

d. Chen, D., Wang, D., Darrell, T., & Ebrahimi, S. (2022). Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 295-305).

e. Wang, D., Shelhamer, E., Liu, S., Olshausen, B., & Darrell, T. (2020). Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726.

f. Yuan, L., Xie, B., & Li, S. (2023). Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15922-15932).

g. Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P., & Tan, M. (2022, June). Efficient test-time model adaptation without forgetting. In International conference on machine learning (pp. 16888-16905). PMLR.

3. No qualitative reustls. The authors are suggested to provide some qualitative results when comparing their approach with the baseline approach. Some failure case analysis will be helpful.

4. The performance of the proposed approach is only verified on EPIC Kitchen, generaliyability to other dataset can be an issue.

5. TSNE visualization on the latent space will be helpful to see how the proposed supervision help during the feature learning procedure. The authors could visualize the changes for different epoches compared with its baseline.

Meta-review: The rebuttal provided clarifications about the proposed method and its analysis that were useful for assessing the paper's contribution and responded adequately to most reviewer concerns. All reviewers recommend acceptance after discussion (with four marginally above the acceptance threshold), and the ACs concur. The final version should include all reviewer comments, suggestions, and additional clarifications from the rebuttal.

Decision: Accept (Poster)