Probabilistic Conformal Distillation for Enhancing Missing Modality Robustness
Authors: mengxi Chen
Fei Zhang
Zihua Zhao
Jiangchao Yao
Ya Zhang
Yanfeng Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Robust Learning
Multimodal Learning
Missing Modality
Primary area: machine_vision
TL;DR: We propose a PCD method to handle the missing modality problem, which transfers privileged information of modality-complete representation by considering the indeterminacy in the mapping from incompleteness to completeness.

Abstract: Multimodal models trained on modality-complete data are plagued with severe performance degradation when encountering modality-missing data. Prevalent cross-modal knowledge distillation-based methods precisely align the representation of modality-missing data and that of its modality-complete counterpart to enhance robustness. However, due to the irreparable information asymmetry, this determinate alignment is too stringent, easily inducing modality-missing features to capture spurious factors erroneously. In this paper, a novel multimodal Probabilistic Conformal Distillation (PCD) method is proposed, which considers the inherent indeterminacy in this alignment. Given a modality-missing input, our goal is to learn the unknown Probability Density Function (PDF) of the mapped variables in the modality-complete space, rather than relying on the brute-force point alignment. Specifically, PCD models the modality-missing feature as a probabilistic distribution, enabling it to satisfy two characteristics of the PDF. One is the extremes of probabilities of modality-complete feature points on the PDF, and the other is the geometric consistency between the modeled distributions and the peak points of different PDFs. Extensive experiments on a range of benchmark datasets demonstrate the superiority of PCD over state-of-the-art methods. Code is available at: https://github.com/mxchen-mc/PCD.

Probabilistic Conformal Distillation for Enhancing Missing Modality Robustness
Mengxi Chen 1.3 Fei Zhang'
Zihua Zhao'
Jiangchao Yao 1.3t Ya Zhang 2.3, Yanfeng Wang
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University
2 School of Artificial Intelligence, Shanghai Jiao Tong University Shanghai Artificial Intelligence Laboratory {mxchen_ ferenas, sjtuszzh, Sunarker, ya_zhang, wangyanfeng))ssje edu.
Abstract
Multimodal models trained modality-complet data are plagued with severe performance degradation when encountering modality-missinng data. Prevalent cross mooss-modal knowledge distillation-be based methods precisely align the represen- tation of poodality-missine data and that of its modality-complet counterpart to enhance robustness. However, due to the irreparable information asymmetry, this features
determinate alignment is too stringent, easily inducing capture spurious factors erroneously. In this paper, novel multimodal Prob- abilistic Conformal Distillation (PCD) method is proposed, which considers the inherent indeterminacy in this alignment. Given a modality-missing input, our goal to learn the unknown Probability Density Function (PDF) of the mapped variables in the modality-complete space, rather than relying on the brute-force point alignment. Specifically. PCD models the modality modality-missing missing feature as a prob- abilistic distribution, enabling it to satisfy two characteristics of the PDF. One the extremes of probabilities of modality-compler feature points on the PDF, and the other the geometric consistency between the modeled distributions and the peak points of different PDFs. Extensive experiments range of benchmark datasets demonstrate the superiority of PCD over state-of-the-ar methods. Code is available at: http:///itub.cmm///ccnenmnnmooo/
1 Introduction
Classical multimodal learning [29, 20, 36, 3] typically pre-supposes that the modalities of all data complete throughout both the training and testing. However, due collection constraints such as device limitations, budget constraints, and restrained working conditions, it is challenging to guarantec such a perfect condition [47]. When modalities are partially available, the performance of models trained on modality-complee data will deteriorate remarkably. This thereby attracts range of explorations contributed recently, given that multimodal learning playing an increasing role.
The existing approaches to address this problem generally fall into two paradigms, i.e., independent modeling [11, 1, 39, 7] and unified modeling [9, 13, 46] for different modality- modality-missing combinations, of which the latter is preferred due to the merits of low-storage cost and flexibility. As one prevalent line of unified modeling, cross-modal knowledge distillation (KD) has achieved persistent advancements in recent years [40, 51, 47, 46). attempts to guide the modality-missing representation to align with its modality-complete counterpart, facilitating the training under the guidance of privileged modality-complet information. However, these methods fail to consider that once a modality is missing, it is impossible to recover its personalized information via a brute-force alignment, which
The corresponding author is Jiangchao Yao (Sunarkerds tu. edu. en).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

has been revealed theoretically by [18]. Roughly ignoring this inherent information asymmetry in the alignment can instead lead multimodal models fit spurious factors erroneously.
We conjecture that when partial modali- ties missing, the retaining information is merely correlated to that of modality- complete input in probabilistic sense. Specifically, given modality-missing in- put, the unknown Probability Density Func- tion (PDF) of its mapped variables in the modality-complete space peaks at the cor- responding modality-complete feature and diminishes when diverging away from this point, as illustrated in Figure 1 (b). Com- pared to previous deterministic methods, learning the PDF is more reasonable and tol- erant way to transfer privileged information. Although the closed form of the oracle PDF is unknown, we can approximate by model- ing probabilistic distribution with two key characteristics: (1) a modeled distribution, the positive points closer to the moodality-complee representation should demonstrate high probabilities and the negative points farther away should exhibit low probabilities. (2) For different distributions from distinct samples, the relation of their peak points should be conformal with that of their modality-complett representations. Here, the former focuses on extreme probability points, while the latter ensures geometric consistency.
K/ Modality 12 Represention
Moddell-Compleee Reprasentation
In a two-modality scenario, when both modalities are present, the modality-complete repre- sentation is derived through fusion. When one modal- ity is absent, the mapped representation inferred from the remaining modality is subject to a certain proba- bility distribution in the modality space.
Figure 1:
With the above intuition, propose novel multimodal Probabilistic Conformal Distillation (PCD) method, which aims to align the modality- missing feature with its probabilistically. Specifically, PCD parameterizes each modality-missing representation as an inde- pendent probabilistic distribution and optimizes it to satisfy the two characteristics. To achieve (1), the log probabilities of the distribution maximized positive points and minimized negative points. To achieve (2), PCD introduces a contrastive-laariin learning-based based approach to align the geometric structure of the peak points of distributions with that of the way, the modeled modality-missing distributions can approximate their corresponding PDFs, thereby facilitating the privileged modality-complet information transfer more efficiently.
counterpart
features. In this
In a nutshell, our contributions can be summarized as follows:
We propose multimodal Probabilistic Conformal Distillation method to handle the missing modality problem, which transfers privileged information of modality-compler representa- tion by considering the indeterminacy in the mapping from incompleteness to completeness.
We parameterize different podality-miising representations as distinct distributions to fit their unknown PDFs in the considering the probabilities of extreme points and ensuring the geometric consistency between peak points of different PDFs and modeled distributions.
space. This is specially realized by
We conduct comprehensive experiments to demonstrate the effectiveness of PCD across a range of mailsigg scenarios. Extensive comparison multimodal classification and segmentation tasks consistently validate the superior performance of our method compared to the state-of-the approaches. Particularly. PCD achieves average improvement of about 5% for the seven podallty-missingg scenarios on the classification dataset CeFA.
2 Related Work
We roughly categorize recent explorations to improve the missing modality robustness into two paradigms: independent modeling methods and unified modeling methods.
2.1
Independent Modeling for Missing Modality
Many works address the modslity-missing problem by training specific models for different modality- missing combinations [41, 10, 31, 26). In certain poddality-misiing case, some approaches recon-

---

struct the original data of the missing modalities from the available ones [2, 22, 28, 28]. However, the complexity of the data reconstruction usually leads to instability and may introduce noise to affect the main task [30, 52]. To alleviate this problem, many works try to reconstruct missing modalities at the representation level [11, 39, 7, 5]. Nevertheless, training specific models for each missing case tend to be inflexible and storage-consuming for real-world scenarios.
2.2 Unified Modeling for Missing Modality
Recently, there has been growing interest in improving the robustness of unified multimodal models against a range of modality-missing combinations [56, 33, 21, 19]. To achieve this goal, some methods attempt to extract redundant information across modalities by designing different fusion networks [15, 53, 9, 50]. However, these methods ignore the complementary information, resulting in suboptimal performance to the specific models. Other methods capture the comprehensive information through dynamical fusion strategies [13, 14, 12, 6]. To be specific, these methods utilize uncertainty estimation techniques to learn the dynamical strength relationships among modalities within different samples, allowing for the adaptive assignment of weights to each available modality. To harness both redundant and complementary information of available modalities more effectively, some methods [32, 51, 47, 46] introduce a distillation loss to guide the unified model to imitate representations inter-sample relations of the modality-complete model. This distillation process help the unified model acquire additional privileged information from complete modalities, so as to improve multimodal robustness [44, 43, 45, 42]. However, previous KD-based methods often emphasize precisely aligning the modality-missing representation with its complete counterpart, which probably causes the overfitting on spurious features due to the inherent information asymmetry.
Method
3.1 Preliminary
Notations.
Suppose that we have a complity training set of {(x), W)) where each M
input x comprises M modalities, denoted as x; = {x;" Jm=1, and y, represents the corresponding ground-truth label. N is the dataset size. Our goal to train unified model capable of accurately predicting the label ys for any modality-missing case x, C & X, # Ø. Here, we use an auxiliary indicator vector ò; for X,, where Vm, 6TTh E {0, 1} indicates the modality x; missing or not. During testing, we construct different modality-missing cases to comprehensively evaluate the robustness.
and modality- missing representations cannot be perfectly aligned, even with redundant information. This claim is experientially supported by the results in Appendix D. Therefore, we try align the representation of modality-missine shown in the right panel of Figure 1, conjecture that the representation 2; of modality-missing input x, has a probabilistic peak expectation towards the representation 2,- of the modality-complet input x In other words, the corresponding PDF p(z
Motivation. Owing to the inherent information asymmetry,
input x; with that of modality-comple
input x; in a probabilistic sense. As
(x.) satisfies the following requirement
arg max p(2(xx). 2
(1)
where Z denotes the representation space. Generally, approximating the unknown PDF p(2/xx, is more relaxed condition compared with the stringent point alignment in previous KD-based methods.
3.2
Probabilistic Conformal Distillation
3.2.1
Objective
Although p(z;(x;) is unknown, even about the function family of the distribution, we can define an easier distribution q(2 (x;) to approximate its characteristics. Specifically, we can force ((2; xx)) to follow the two-fold characteristics: 1) extremum property. In modeled distribution q(2 representation 2 should exhibit higher probabilities, and negative points distant from 2 approach far smaller probabilities. different samples, the relationship of the peak points of ((=|x) should be conformal with that of their corresponding modality-complet points 2".
positive points near the modality-compler
(2) conformal property: Given
3

---

Teacher
Probability Extremum
Geometric Consistency
(0)
2 0
max
PCD
min
g
g
Student F (0)
Student F (1)
Zp
Scalar Multiplication
Missing Modality Indicator
Pulling Together
Figure 2: An overview of the proposed method. PCD self-KD architecture, where the teacher and student share the same framework. The teacher provides the modality feature 2" and the geometric structure g" guide the student. In the student, modality-missing features are parameterized as different normal distributions to fit the corresponding PDF. To achieve this, PCD maximizes distributions at positive z/ and minimizes 2,, while aligning g with positive g".
To achieve the former property, we first define a positive set Zp which includes all coodality-complee representations zp that are close to 27, and negative set Zn consisting of the remaining representa- tions =, that are far away from 2/. For example, in classification task, Zp contains all 2 of the same class as X,, while Zn consists of = from other classes. In segmentation, Zp contains only 2/. Then, the following characteristic should be satisfied
q( E Zp|x;) > q(2, Zn|x;)
(2)
Equation (2) encourages that the probability of any one positive point zp to be greater than the probabilities of all negative points 2, Zn. which helps to satisfy the extremum property.
Regarding the conformal property, let 9i denote the geometric vector for 2;. Each element in 9: calculates the distance between the peak points of (2, x2) and other modeled distributions q(2, (xy) Vector 9* represents the geometric distance calculated by the modality compllet-- representations 2" in the same manner. Similar Z,, and Z,, we use G, and G, include the positive and negative geometric vectors respectively. The set Gp contains all the vectors 93 corresponding to zØ and the same relation applies Gn and 2/, Then pursue the following characteristic satisfied
s(g E Gp. 91)
s(g Gn.gt),
where .(.,)) )) > 0 one the metrics for measuring the vector similarity. Equation (3) hopes the similarity between the geometric vector Ii and any positive vector g" be larger than that between Ii and negative vectors 9*. E Gn- To meet the characteristics Equation (2) and Equation (3), we propose a probabilistic conformal objective optimize (2[xx)):
Ng:ee, $ 9:) II,e2, 4 (2)11x;)
max
11, ((2) xx)
Specifically in Equation (4), to satisfy the extremum property, we propose to maximize the proba- bilities of (22z/1xx (xx) at positive points = E Zp and minimize them at negative points 2 E Zn. To achieve Equation (3), we introduce contrastive learning-based approach maximize the similarities s(g,,9) 9:). Notice that, here the minimization of s 9i) not emphasized, since it is implicitly included in the contrastive-leamins bbased based similarity. By simplifying Equation (4) with the log function, we can transform the objective function into a more manageable form, expressed as
max
Σ E
log ((>))xx))
Σ Z.
log (25)(x)))
Σ SE
log s 96)
(5)
Probability Extremum
Geometric Consistency
Equation (5) consists of two parts, where the first term focuses on extreme probability points, while the second term is for the geometric consistency. In the following, we introduce the implementation of Equation (5) on how to model the modality-missing distributions (Section 3.2.2) and fit the corresponding PDFs (Section 3.2.3).

---

3.2.2 Multimodal Probabilistic Modeling.
The framework of PCD is shown in Figure 2. For each modality-misse input x, we establish an individual D-dimensional normal distribution g(25[x;), with its mean and variance directly determined as by the multimodal encoder follows
((2x)x))) ~ ~N (i: where μ4 = (
(6)
The features 14; E RD, E RD represent the mean and variance vectors of the multimodal distribution N (u,, of). respectively. ʃ(.) denotes the multimodal encoder, while k(.)) is the head module for computing the variance vectors. We maximize Equation (5) for each modality-missing distribution q(2 xx4) to fit the corresponding PDF. The probabilistic modeling maps each modality- missing input x; to a density region in the representation space, rather than a single deterministic vector point, which enhances the tolerance to lower-quality modality- data and prevents the multimodal encoder from affecting representation capacity by learning some spurious factors.
3.2.3 Probabilistic Conformal Distillation
After modeling the modality-misssinn input as a Gaussian distribution (xx](x;), we aim to approxi- mate ((2//x,) to the unknown PDF p(z; x;) to transfer the codalitt-complete information. This is accomplished by optimizing two terms in Equation (5), that is, the probability extremum term and the geometric consistency term.
The probability extremum term in Equation (5) enables q(2/(x;) to have higher probabilities at positive points in Zp and lower probabilities at negative points in Za- By inserting the Gaussian function into the probability extremum term and eliminating the constant, the extremum term can be maximized by minimizing its negative form, namely, the following loss,
Probability Extremum.
(#p.d mi,a)² 2(0,,a)2
(2 mi,a) + log Fi,d (7)
-- Σ Σ
+ log ozdd
Σ Σ
2(
{plp~=u.) a
Prior works [4, 35] high-dimensional latent distribution learning report that the variance collapse commonly encountered issue. This phenomenon typically occurs because the network is encouraged to predict small a values to mitigate the unstable gradients that arise while using Stochastic Gradient Descent. To prevent this problem, we empirically implement clipping operation on Equation (7), stopping the optimization when a becomes too small. For brevity, we focus on analyzing the first half of Equation (7). Its optimization is carried out in two aspects: (1) minimizing the distance between the mean fi and the positive modality-complee
representations 2pd of the teacher, i.e., (2p,a 2 (2) correlating this distance with of where larger distances correspond to higher variance, and vice versa. This relationship allows us to estimate the element-wiss quality of each mean vector μ4. where the closer proximity to 2 signifies more information contained.
Geometric Consistency. The geometric consistency term aims to align the structure vector 9: with its positive counterparts in Gp. Specifically, we represent the geometric vector g" of PDFs by calculating the distances of their peak points z", and is obtained by the distances of mean vectors μr, namely:
ot (b) =(), (b) = a(Ji, /4b):
where Gis gi are dimmennnsll vectors with P4: 2/ as the cores, respectively. |B] is the batch size. Theoretically, a(.,.) can be any formula for calculating the distance between vectors. For classification tasks, -)::)) the Euclidean distance. For segmentation tasks, since the dimension of the modality-missing and modality-complet product to mitigate the curse of dimensionality. Notice that gi: of are computed across samples in the batch, without distinguishing between positive and negative samples.
features could be very high, we choose the inner
Like Zp. the set G, contains the positive geometrie vectors 9z. whose core z share the same class as X3, namely Gp {gply,
= y:). For the similarity function s(pp 9i) in the geometric consistency term, we employ the following contrastive learning-base bassdd form:
xxp(3(g) 9:)/T)
$ 9:) = exp(3(0,, 9))gg)..////)) + Σ ex)()()t...)/1))
(8)
{ kvl
5

---

where G g²) calculates the cosine similarity between 9 and g², T is the temperature coefficient. It worth noting that the segmentation task, due the high dimensionality of multimodal features, only one negative vector is selected to conserve computational resources. Then, PCD aligns with g" E Gp through minimizing:
Lg =-
Σ log (g)g {plDD=wi
To reiterate, the difference between the contrastive learning-based loss Lg in classification and segmentation tasks analogous that between supervised contrastive learning [23] and contrastive learning [16, 55, 54, 48]. The former considers all g" sharing the same class as as positive samples, whereas the latter only uses 97 from the same instance the positive sample.
By optimizing Equation (7) and Equation (9), each modality-missing distribution can fit the corre- sponding p( p(= and capture privileged information from the teacher in more tolerant way.
3.3 Training Process
The framework of PCD, depicted in Figure 2, adopts a teacher-student architecture. Self-KD [24] is introduced to build an end-to-end distillation system, where the parameters of the fixed teacher F(0) are obtained from the warm-up stage. During the training stage, the teacher model handles the modality-complete data and provides supervision for the student F()..
Overall Loss. The overall loss function formulated as:
L = L: + X(Ln + Lg);
(10)
where the hyperparameter used to balance different losses, and the experiments show that À is insensitive in a certain range. L, represents the task learning loss, which is defined by the specific primary task. For example, when the primary task is classification, Li corresponds to the cross-entropy loss. The training procedure is shown in Algorithm 1 in Appendix A.
3.4 Discussion
PCD proposes to fit the PDFs of variables in the representation space by utilizing different parameter- ized Gaussian distributions. Compared to existing KD-based methods, PCD offers more tolerant and reasonable way to transfer the privileged information from the modality-comple the modality-missine
teacher to student. Specifically, it optimizes the probabilities of modeled distributions at extremum points and constrains the alignment between the geometric structures of teacher repre- sentations and the mean vectors of modeled distributions. Besides, regarding the complexity, PCD only introduces some head modules in the encoder to estimate the variance, which is lightweight and efficient and can be easily applied to many existing multimodal fusion methods.
4 Experiments
4.1 Experimental Setup
Datasets. We implement experiments four multimodal datasets, comprising two classification datasets CASIA-SURF and CeFA, and two segmentation datasets NYUv2 and Cityscapes.
CASIA-SURF [49] and CeFA [27] are two large face anti-spoofing datasets that include samples across three modalities: RGB, Depth and infrared (IR). For CASIA SURF [49], we adhere to the intra-testing protocol established by the authors, ensuring consistency and reliability in our experimental results. This dataset comprises 29,000 samples for training, 1,000 for validation, and 57,000 for testing. Similarly, in CeFA [27]. we employ a cross-ethnicity and cross-attack protocol as recommended by the authors, which divides the dataset into training, validation, and testing sets with 35,000, 18,000, and 54,000 samples respectively.
NYUv2 [37] and Cityscapes [8] are both two-modality segmentation datasets, each comprising RGB and Depth modalities. NYUv2 [37] contains total of 1 1,449 indoor RGB-D images, with 795 designated for training and 654 for testing. NYUv2 employs a common 40-class label setting, facilitating comparative analysis across various segmentation algorithms. Cityscapes [8] is outdoor

---

Table I: Performance under different multimodal conditions, where "R", "D", and "I" respectively represent the available RGB, Depth, and IR modality. "Average" is the average performance over all the possible conditions. ACER + means that the lower the ACER value, the better the performance, while mIOU f is the opposite. The best results are bold and the second-best ones are marked with underline. "A" means the performance gap between PCD and the best results.
CASIA-SURF
(ACER U)
Method
[R]
(D)
[1]
(R.D)
[R.I]
[D.I]
[R.D.I)
Average
Traditional [49]
23.03
17.10
49.53
10.40
41.02
11.26
1.40
22.11
Sepurate Model [49]
10.01
4.45
11.65
3.41
6.32
3.54
1.23
5.80
Augmentation [1]
11.75
5.87
16.62
4.61
6.68
4.95
2.21
7.52
HeMIS [15]
14.36
4.70
16.21
3.23
6.27
3.68
1.97
7.18
MMFormer [50]
11.15
4.67
13.99
1.93
4.77
3.10
1.94
5,93
MMANET [46]
8.57
2.27
10.04
1.61
3.01
1.18
0.87
3.94
MD [12]
10.84
6.65
19.43
12.64
7.84
3,99
0.96
7.30
ETMC [14]
7.91
4.73
7.54
1.39
4.56
1.46
0.76
4.05
RAMI. [6]
11.26
3.10
11.65
1.92
5.35
1.76
1.09
5.16
PCD
7.23
2.20
5,66
0.99
2.86
0.89
0.74
2.93
A
0.74%
0.07%
188%
0.40%
0.15%
0.29%
0.02%
1.01%
CeFA (ACER L)
Method
(R)
(D)
(1)
(R.D)
[R.1]
(D.I)
(R.D.I)
Average
Traditional [49]
50.00
50.00
49.96
49.25
47.28
48.95
39.62
47.86
Sepurate Model [49]
27.44
33.75
36.17
35.62
31.62
36.62
24.15
32.20
Augmentation [1]
27.93
36.90
36.14
32.10
28.47
35.12
31.87
32.65
HeMIS [15]
34.14
37.97
36.94
36.02
33.94
31.92
40.66
35.94
MMFormer [50]
28.51
33.58
39.56
29.47
27.66
32.17
30.72
31.52
MMANET [46]
27.15
32.50
35.62
22.87
23.27
30.45
23.68
27.94
[12]
27.13
37.99
26.25
34.69
31.95
ETMC [14]
24.74
34.28
37.62
22.52
24.25
30.63
21.59
27.95
RAML [6]
28.54
33.88
40.01
23.82
28.81
28.85
22.11
29.43
PCD A
21.38
28.01
34.79
17.19
20.92
21.68
14.39
22.63
3.36% 1
4.49%
0.83%
5.33%
2.35%
5.75%
7.20%
5.31%
NYUv2
(mIOU 1)
Cityscapes
(m1OU f) (R.D)
Method
(R]
(D)
(R.D]
Average
(R)
(D)
Average
Traditional [36]
11.15
4.18
48.78
21.41
3.17
4.87
78.73
28.89
Separate Model [36]
44.22
40.55
48.89
44.55
77.60
59.11
78.62
71.77
Augmentation [1]
41.34
39.76
47.23
42.77
76.89
57.42
78.13
70.81
MMFormer [50]
43.22
41.12
48.45
44.26
76.62
58.53
78.01
71.05
MMANET [46]
44.93
42.75
49.62
45.58
77.61
60.12
78.89
72.20
PCD
45.68
44.34
49.44
46.49
78.26
61.30
79.53
73.03
A
0.75% 1
1.59%
0.18%
0.91%
0.65%
1.18%1
0.64%
0.83% 1
RGB-D dataset designed for urban scene comprehension. There are 5,000 annotated samples, where 2,975 samples are for training, 500 for validation, and 1,525 for testing.
Experimental Details. For classification CASIA-SURF and CeFA, the SGD optimizer [34] is used and the batch size is 64. The dimension of the Gaussian distribution is 512. We report the results using the metric of Average Classification Error Rate (ACER). Each modality leverages separate ResNet-18 [17] as the unimodal encoder. We employ an exponential decay learning rate strategy in which the learning rate fixed at 1e-3 during the warm-up stage and then decays exponentially. Weight decay and momentum are set to 0.0005 and 0.9, respectively. For segmentation experiments use the Adam optimizer [25] and set the batch size 16. The
on NYUv2 and Cityscapes, results are evaluated by the metric of mean IOU (mIOU). The learning rate is initialized with le-2 and le-4 respectively for two datasets and adapted by the one-cycle scheduler. Following [46], we use ESANet [36] as the backbone. On all datasets, the variances are obtained through two-layer MLP, where the hidden size 1024. During training, we augment each simulating all potential modality-missing scenarios and randomly sample one of the augmented data as the training sample for the current epoch. For bimodal datasets, three cases are included, that is, missing RGB, missing depth, and complete. For trimodal datasets, there are seven missing cases.
data by

---

4.2 Performance Comparison
To evaluate the robustness of PCD, we choose the following methods in the comparison: 1) Baselines. Traditional [49, 36]: a benchmark method trained solely on
data. Separate
Model [49, 36]: separate intermediate- fusion models for each modality combination. 2) Redundancy- based methods: Augmentation [1], MMFormer [50]. 3) Cross-modal KD-based methods: MMIN [51],
MMANET [46]. 4) Dynamical fusion-based methods: MD [12], ETMC [14), RAML [6].
Classification Task. The results in Table I show the performance of PCD and other state-of-the-an (SOTA) methods across various testing conditions with missing modalities on two classification datasets CASIA-SURF and CeFA. We can see that the 'Traditional" method, which is exclusively trained on cedaality-copller samples, exhibits a high sensitivity to the missing modality problem. Specifically, the enror rate surges by 21.63% on CASIA-SURF when only the RGB modality is available. Comparing the results of various missing modality methods. PCD achieves the best results in almost all the settings on the two multimodal classification datasets. In comparison to the second-best method, PCD demonstrates the error rate reductions of 1.01 % and 5.31 % on CASIA- SURF and CeFA. These results illustrate the effectiveness of our proposed method in privileged information transfer. Besides, the performance of some methods declines with an increasing number of modalities. For example, on CeFA, the error rate of MMANET with complete modalities is 0.81 % higher than when IR is absent. This deterioration may potentially caused by overfitting resulting from deterministic alignment. In contrast, our method employs probabilistic distillation, which introduces a more relaxed framework for aligning representations, mitigating this issue effectively.
Segmentation Task. We conducte experiments on NYUv2 and Cityscapes to verify the effectiveness of PCD on segmentation tasks. Compared to the second-best method, PCD achieves average accuracy improvements of 0.91% and 0.83% on NYUv2 and Cityscapes, respectively. Furthermore, in the Depth-missing scenarios on the NYUv2 and Cityscapes datasets, PCD demonstrates relatively small improvements. This may be because that the performance of the input RGB is already very close to that of the modality-complete input. Consequently, is challenging to obtain additional privileged information through distillation, limiting the potential enhancement.
4.3 Further Analysis
Ablation on Loss Components.
Table 2: Ablation study on CeFA. and v in the table indicate without and with the corresponding loss term respectively.
In this part, investigate the impact of each loss component in Eq. (10) on CeFA. In Table 2, we conduct the ablation study and summarize the correspond- ing performance with or without different loss components. Ac- cording to the results in Table 2, we can observe that the classi- fication model with the probability extremum loss Eu performs 3.36% better than the sim- ple model with only Le, which suggests that constraining probabilities of extreme points in- deed helps to the privileged information transfer from the cedality-complet teacher to the modaalty-mmising student. Additionally, PCD with all loss components outperforms the model with L. and En on average, which validates the effectiveness of the geometric consistency loss.
Le Cu
C.
(R)
(D)
(1)
(R.D)
(R.I)
(D.1)
(R.D.I) Average
v
26.95
38.06
37.06
24.18
24.75
32.82
25.38
29,89
V
V
21.14
33.76
37.22
21.28
23.61
27.56
21.19
26.53
V
20.62
34.43
35.23
18.18
21.86
32.63
21.72
26.38
v
21.38
28.01 34.79
17.19
20.92
21.68
14.39
22.63
Ablation Probabilistic Dis- tillation. To study the effect of probabilistic distillation, we conduct experiments to compare the performance of PCD with its determinate distillation variant. Here, the variant is the degrada- tion method of PCD that trans- fers knowledge by directly mini- mizing the Euclidean distance of the complete-incomplete pairs in teacher and student networks. The results are shown in Table 3. It can be seen that PCD consistently
Table 3: The comparison between PCD and its variants on CeFA, where "Determinate" means the degradation of PCD with deter- minate distillation, while "Pretrained" is the distillation with pretrained teacher.
Configurations
[R) (D)
[1]
(R.D) (R.I)
(D.I)
(R,D.I)
Average
Determinate
23.52 38,96 38.95
25.75
24.52
36.1
28.21
30.99
Pretrained
23.52
31.64
39.86
22.57
24.89
29.43
26.50
28.34
PCD
21.38
28.01
34.79
17.19
20.92
21.68
14.39
22.63
8

---

outperforms its "Determinate" variant in all missing modality combinations and decreases the error rate by 8.36% on average. This demonstrates the effectiveness of transferring privileged information via probabilistic distillation, which is more tolerant.
Analysis about KD Strategy. To explore the effectiveness of self-KD,, compare PCD with its pretrained teacher variant. This variant refers to training modality-compler teacher individually to guide students in optimizing from scratch. The results are shown Table 3. As can be seen, the error rate of PCD is 5.71 % lower on average than its pretrained variant. In addition to training fixed teacher to offer coopleee supervision, our self-KD strategy also provides good initialization for the student. With the help of the shared predictor, the semantic coherence of modality- complete and modality- missing representations is indirectly ensured, which narrows information gap between them at the beginming of KD, thereby facilitating privileged information transfer.
Classification Boundary of the Teacher and Student. In order to further validate the effectiveness of probabilistic distilla-
Class
300
Sample Num
240
tion for the transfer of privileged complete-
180
modality information, we analyze the pre-
120
10
10
dictions of both the fixed teacher obtained from the warm-up stage and the distilled student under all multimodal conditions. The results are shown Figure It can be observed that, apart from the reduced error rate, the logits of the student exhibit a higher concentration around 0 or 1, demon- strating more separable inter-class bound- ary. The probabilistic distillation process transfers privileged information to hard samples around the classification boundary in more tolerant way, mitigating the erroneous fit to spurious factors, so as to further refine modality-missiin features.
0.0
0.2
0.4
0.8
1.00.0
0.2
0.6
0.8
1.0
(a) Warm-up
(b) MPCD
The prediction distributions of both the teacher and the distilled student of PCD under all mul- timodal combinations on CeFA. The X-axis represents the normalized logit output and the Y-axis the number of samples after taking the square root.
Figure 3:
Hyperparameter X. The hyperparameter controls the balance between distillation and classification. To validate the stability of PCD against X, we conducted several experiments with different values of À on CeFA. The results are shown the left half of Figure 4, where values of À range from 1.4 to 2.4. From the curve, we can see that setting relatively large value for X enhances the distillation of privileged infor- mation, thereby enhancing the multimodal robustness. Specifically, in [1.4, 2.2], X appears be insensitive within certain range, In our experiments, we À 1.8.
25.3
24.2
23.1
22.0
20.9
1.6
1.8 A
2.0
2.2 0.2
0.5
T
Figure The average performance of PCD under dif- ferent À and values on CeFA. The hyperparameter X is used to balance the loss terms, 7 the temperature.
Hyperparameter т. In the right panel of Figure 4, we conducted several experiments with different values of 7 to assess its impact our results. The hyperparameters is the temperature in Equa- tion (8), which scales the similarity measures. The results reveal T is insensitive within a certain range. In our experiments, we set 0.5.
Computational Overhead. Compared to the multimodal models with the same backbone, PCD only introduces a few additional head mod- ules in the encoder to estimate the variance. To demonstrate the minor change PCD brings, we estimated the number of parameters and FLOPs of PCD and the other three late fusion methods in Table 4. It can be seen that PCD does not significantly increase the number of parameters or FLOPs, where the FLOPs are almost equal to
Table 4: The numbers of parameters (M) and FLOPs (G) of several methods on CeFA.
Method
Backbone
Paramters
FLOPs
MD [12]
ResNet-18
35.88
1.392
ETMC [14]
ResNet- 18
34.30
1.391
RAML [6]
ResNet- 18
35.09
1.393
PCD
ResNet-18 18
38.50
1.395

---

the second-best method MMANET, while the number of parameters only increased by 4.20M. This lightweight change of MPCD makes it easily be applied to many existing multimodal fusion methods.
Table 5: Performance under different multimodal conditions when each unimodal data of training samples is missing with probability of 30%.
Method
(R]
(D)
[1]
[R.D]
(R,1)
[D.I]
(R,D.I)
Average
MMANET [46]
28.39
39.61
34.12
34.19
23.39
34.12
27.11
31.56
ETMC [14]
25.96
34.69
38.60
24.15
24.58
31.83
24.03
29.12
PCD A
23,42
30.23
34.60
18.34
21.98
24.50
15.07
24.02
2.54%
4.46%
0.48%
5.81%
1.41%
6.43%
8.96%
5.20%
Modality-Missingg Training Data. All the experiments above are conducted with the modality- complete training data. In this part, we extend PCD by considering the scenario where the modality- complete data of some training samples is also unavailable. PCD is only applied to the data that has modalittyccmpllet counterpart, and for the remaining data, only L. is optimized. Here, we introduce a case where 30% of the data is consistently missing from each modality during training. As shown in Table 5, while some modality-missine outperforms the second- best method by 5.20% on average. Although PCD is not specifically designed for modality-missingg training data, these results demonstrate its scalability for such scenarios.
cases may underperform compared the SOTA, PCD still
Conclusion
In this paper, we propose a multimodal Probabilistic Distillation (PCD) method to mitigate the missing modality problem, which considers the indeterminacy in the aligmment between the modality and modality representations. Specifically, PCD aims to parameterize the modality-missin representations as different Gaussian distributions and fit PDFs of their mapped variables in the modality-complee space. This is achieved by ensuring the characteristics of probabilities at extreme points and maintaining geometric consistency with that of the experiments validate the superiority of PCD in increasing multimodal robustness.
features. Extensive
Acknowledgement
This work is supported by the National Key R&D Program of China (No. 2022ZD0160702), STCSM (No. 22511 155111066 106101, No. 22DZ2229005), 111 plan (No. BP0719010) and National Natural Science Foundation of China (No. 62306178).
References
[1] Michal Bednarek, Piotr Kicki, and Krzysztof Walas. On robustness of multi-modal fu- sion-robotics perspective. Electronics, 9()):115,, 2020.
[2] Lei Cai, Zhengyang Wang, Hongyang Gao, Dinggang Shen, and Shuiwang Ji. Deep adversarial learning for multi modalty missing data completion. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. pages 1158-1166, 2018
[3] Jinming Cao, Hanchao Leng, Dani Lischinski, Daniel Cohen Or, Changhe Tu, and Yangyan
Li. Shapeconv: Shape-aware convolutional layer for indoor rgb-d semantic segmentation. In Proceedings of the IEEF/CVF international conference on computer vision, pages 7088-7097, 2021.
[4] Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 7710-5519, 2020.
[5] Mengxi Chen, Linyu Xing, Yu Wang, and Ya Zhang. Enhanced multimodal representation learning with cross-modal kd. Proceedings of the IEFF/VV Conference on Computer Vision and Pattern Recognition, pages 11766-11775, 2023.
10

---

[6] Mengxi Chen, Jiangehao Yao, Linyu Xing, Yu Wang, Ya Zhang, and Yanfeng Wang. Redundancy-adden adaatiiv multimodal learning for imperfect data, arXiv preprint arX1:2330..4496. 2023.
[7] Yanbei Chen, Yongqin Xian, A Koepke, Ying Shan, and Zeynep Akata. Distilling audio-vist visual knowledge by compositional contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7016-7025, 2021.
[8] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[9] Yuhang Ding, Xin Yu, and Yi Yang. Rfnet: Region-aware fusion network for incomplete multi- modal brain tumor segmentation. In Proceedings of the IEEE/CVF international conference computer vision, pages 3975-3984, 2021.
[10] Tiantian Feng, Daniel Yang, Digbalay Bose, and Shrikanth Narayanan. Can text-to-image model assist multi-modal learning for visual recognition with visual modality missing? arXiv preprint arXiv:2402.09036 2024.
[11] Nuno C Garcia, Pietro Morerio, and Vittorio Murino. Modality distillation with multiple stream networks for action recognition. In Proceedings of the European Conference Computer Vision (ECCV), pages 103-118, 2018.
[12] Zongbo Han, Fan Yang, Junzhou Huang, Changqing Zhang, and Jianhua Yao. Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification. In Proceedings the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 20707-207 17, 2022.
[13] Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification. arXiv preprint arXiv:2102.02051, 2021.
[14] Zongho Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification with dynamic evidential fusion. IEEE transactions on pattern analysis and machine intelligence, 45(2):2551-2566, 2022.
[15] Mohammad Havaei, Nicolas Guizard, Nicolas Chapados, and Yoshua Bengio. Hemis: Hetero- modal image segmentation. In Medical Image Computing and Computer-Assisted Intervention- MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceed- ings, Part II 19, pages 469-477. Springer, 2016.
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2020.
[17] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances Neural Information Processing Systems, 34:10944-1000 10956, 2021.
[19] Ziqi Huang, Li Lin, Pujin Cheng, Linkai Peng, and Xiaoying Tang. Multi-modal brain tumor segmentation via missing modality synthesis and modality-level attention fusion. arXiv preprint arXiv:2003.0 04586, 2022.
[20] Wen-Da Jin, Jun Xu, Qi Han, Yi Zhang, and Ming-Ming Cheng- Cdnet: Complementary depth network for rgb-d salient object detection. IEEE Transactions on Image Processing, 00:3376-3390, 2021.
[21] Vijay John and Yasutomo Kawanishi. A multimodal sensor fusion framework robust to missing modalities for person recognition. In Proceedings of the 4th ACM International Conference on Multimedia in Asia, pages 1-5, 2022.
11

---

[22] Jiang Jue, Hu Jason, Tyagi Neelam, Rimner Andreas, Berry L Sean, Deasy O Joseph, and Veer- araghavan Harini. Integrating crossmmodality hallucinated mri with et to aid mediastinal lung tu- mor segmentation. In Medical Image Computing and Computer Assisted Intervention-MICe MMCCAAI 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part VI 22, pages 221-229. Springer, 2019.
[23] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33: 13:1116611877, 2020.
[24] Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distillation with progressive refinement of targets. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2021.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11Xiv:1412.6980, 2014.
[26] Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen- Yu Lee. Multimodal prompting with missing modalities for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14943-14952, 2023.
[27] Ajian Liu, Zichang Tan, Jun Wan, Sergio Escalera, Guodong Guo, and Stan Li. Casia-surf cefa: A benchmark for multi-modal cross-ethnicity face anti-spoofing. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 1179-1187, 2021.
[28] Ajian Liu, Zichang Tan, Jun Wan, Yanyan I Liang, Zhen Lei, Guodong Guo, and Stan Z. Li. Face anti-spoofing via adversarial cross-modality translation. IEEE Transactions on Information Forensics and Security, 16:2759-2772, 2021.
[29] Ajian Liu, Jun Wan, Sergio Escalera, Hugo Jair Escalante, Zichang Tan, Qi Yuan, Kai Wang. Chi Lin, Guodong Guo, Isabelle Guyon, et al. Multi-modal face anti-spoofing attack detection challenge at evpr2019. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[30] Haojie Liu, Shun Ma, Daoxun Xia, and Shaozi Li. Sfanet: A spectrum- aware feature augmenta- tion network for visible-infrared person reidentification. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[31] Mengmeng Ma. Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust to missing modality? In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18177-18186, 2022.
[32] Mengmeng Ma. Jian Ren, Long Zhao. Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2302-250 2310, 2021.
[33] Harsh Maheshwari, Yen-Cheng Liu, and Zsolt Kira. Missing modality robustness in semi- supervised multi-modal semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1020-1030, 2024.
[34] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
[35] Enrique Sanchez, Mani Kumar Tellamekala, Michel Valstar, and Georgios Tzimiropoulos. Affective processes: stochastic modelling of temporal context for emotion and facial expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9074-9084, 2021.
[36] Daniel Seichter, Mona Köhler, Benjamin Lewandowski, Tim Wengefeld, and Horst-Michael Gross. Efficient rgb-d semantic segmentation for indoor scene analysis. 202/ IEFE interna- tional conference robotics and automation (ICRA), pages 13525-13531. IEEE, 2021.
12

---

[37] Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi multi-domain operations applications, volume 1 1006, pages 369-386. SPIE, 2019.
[38] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A Agb-d scene understand- ing benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567-576, 2015.
[39] Jonathan Stroud, David Ross, Chen Sun, Jia Deng, and Rahul Sukthankar. D3d: Distilled 3d networks for video action recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 625-634, 2020.
[40] Luan Tran, Xiaoming Liu, Jiayu Zhou, and Rong Jin. Missing modalities imputation via cascaded residual autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1405-1414, 2017.
[41] Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carneiro. Multi- modal learning with missing modality via chared-speciff feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 15878-15887, 2023.
[42] Hu Wang, Congbo Ma. Yuyuan Liu, Yuanhong Chen, Yu Tian, Jodie Avery, Louise Hull, and Gustavo Cameirr. Enhancing multi-modal learningg Meta-leamndd cross-modal knowledge distillation for handling missing modalities. arXiv preprint arXiv:2405.07155, 2024.
[43] Hu Wang, Congbo Ma, Jianpeng Zhang, Yuan Zhang. Jodie Avery, Louise Hull, and Gustavo Carneiro. Learnable cross knowledge distillation for multi-modal learning with missing modality. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 216-226. Springer, 2023.
[44] Qi Wang, Liang Zhan, Paul Thompson, and Jiayu Zhou. Multimodal learning with incomplete modalities by knowledge distillation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1828-1838, 2020.
[45] Shuai Wang, Zipei Yan, Daoan Zhang, Haining Wei, Zhongsen Li, and Rui Li. Prototype knowledge distillation for medical segmentation with missing modality. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[46] Shicai Wei, Chunbo Luo, and Yang Luo. Mmanet: Margin- aawre distillation and modality- aware regularization for incomplete multimodal learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 20039-20045 2023.
[47] Jiandian Zeng, Tianyi Liu, and Jiantao Zhou. Tag-assisted multimodal sentiment analysis under uncertain missing modalities. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development Information Retrieval, pages 1545-1554, 2022.
[48] Fei Zhang, Tianfei Zhou, Boyang Li, Hao He, Chaofan Ma, Tianjiao Zhang, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Uncovering prototypical knowledge for weakly vocabbacyyylary semantic segmentation. Advances in Neural Information Processing Systems, 36:73652-73665. 2023.
[49] Shifeng Zhang, Xiaobo Wang, Ajian Liu, Chenxu Zhao, Jun Wan, Sergio Escalera, Hailin Shi, Zezheng Wang, and Stan Z Li. A dataset and benchmark for large-scale multi-modal face anti-spoofing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 919-928, 2019.
[50] Yao Zhang, Nanjun He, Jiawei Yang, Yuexiang Li, Dong Wei, Yawen Huang, Yang Zhang, Zhiqiang He, and Yefeng Zheng. mnformer: Multimodal medical transformer for incomplete multimodal learning of brain tumor segmentation. In International Conference on Medical Image Computing and Computer-Assiste Intervention, pages 107-117. Springer, 2022.
13

---

[51] Jinming Zhao, Ruichen Li, and Qin Jin. Missing modality imagination network for emotion recognition with uncertain missing modalities. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1th International Joint Conference on Natural Language Processing, pages 2608-2611 2021.
[52] Zihua Zhao, Mengxi Chen, Tiajjie Dai, Jiangehao Yao, Bo Han, Ya Zhang, and Yanfeng Wang. Mitigating noisy correspondence by geometrical structure consistency learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27381-27390, 2024.
[53] Tongxue Zhou, Stéphane Canu, Pierre Vera, and Su Ruan. Brain segmentation with miss- ing modalities via latent multi-source correlation representation. In Medical Image Computing and Computer Assisted Intervention-MiICCA 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part IV 23, pages 533-541. Springer, 2020.
[54] Zhihan Zhou, Jiangchao Yao, Feng Hong. Ya Zhang, Bo Han, and Yanfeng Wang. Combating representation learning disparity with geometric harmonization. In Thirly-seventh Conference on Neural Information Processing Systems, 2023.
[55] Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference on Machine Learning, pages 27367-27377. PMLR, 2022.
[56] Yizhe Zhu, Xin Sun, and Xi Zhou. Exploiting multi-modal fusion for robust face representation learning with missing modality. In International Conference on Artificial Neural Networks, pages 283-294. Springer, 2023.
14

---

Appendix Supplemmenal Material
A
Algorithm
The whole training procedure of PCD is shown in Algorithm 1.
Algorithm Training procedure of PCD
Training data X, target set Y, missing modality indicator 8, hyperparameters X, T, warm-up epoch E, training epoch P and batch size
Input:
// The warm-up stage
for an epoch I. do for sampled batch X Optimize Co
{x: Y (us) do
Initialize the teacher and student models
end for end for
// The training stage
Load and fix parameters in warm-up stage for the teacher
for epoch p = 1.... do
for sampled batch = - Y {v: do
Calculate =* and g²
Model Gaussian distributions in Equation (6)
Calculate the probability extremum loss Lu and the geometric consistency loss Lag
Optimize Equation (10) to update parameters of the student
end for end for
B Implement Details
B.1
Classification
Network Architecture. For a fair comparison, we follow the basic implementation the traditional multimodal model in [49] for all the comparison methods. This backbone is late fusion network with separate ResNet18 encoders for each modality. Here, for PCD, we parameterize the unimodal features from unimodal encoders and the fused multimodal features from the fusion module as independent Gaussian distributions, and make them their PDFs by optimizing corresponding Cu and Lg. The variance is obtained for two-layer MLP, where the hidden size is 1024. In addition, like [6], by analyzing the variance in unimodal distributions, a weighting mechanism is employed, which can adaptively aggregate the information of each available unimodality.
Setup. We augment modality-complete samples by simulating all potential missing modality see- narios equally. In other words, in one epoch, each sample has an equal probability of randomly encountering one of seven missing modality scenarios. Besides, random flipping, rotation, and cropping are also used for data augmentation. All models optimized by SGD for 110 epochs with a mini-batch 64. Weight decay and momentum are set to 0.0005 and 0.9, respectively. The learning rate is initialized 0.001. After the warm-up stage, exponential decay learning rate strategy is employed, in which the decay coefficient is 0.9. The dimension of the Gaussian distribution is 512. The srpertpprrmetter r are 1.8 and 0.5, respectively.
B.2 Segmentation
Network Architecture. We use the ESANet [36] as the backbone, which is an early fusion network. The modality encoder is the ResNet50 with NBt1 used in ESANet. For PCD, we parameterize the fused multimodal features from the last three resolution stages as independent Gaussian distributions. Notice that, since the dimensionality of multimodal features is very high, only negative vector in Equation (8) is selected to conserve computational resources, and this formulation degenerates to the triplet loss. Besides, C., is applied the fused features after average pooling.
Setup. Random flipping, rotation, cropping and missing modality simulation are used for data augmentation. All models are optimized by Adam for 450 epochs with mini-batch of 16. The
15

---

Table Stability experiments on NYUv2, Cityscapes, CASIA-SURE and CeFA.
CASIA-SURF
[R]
(D)
(1)
[R.D)
[R.I)
(D.I)
[R.D.I)
Average
2.20
5.66
0.99
2.86
0.89
0.74
2.93
PCD
=0.13
+0.26
+0.90
10.10
#0.31
#0.19
#0.23
0025
CcFA
(R)
(D)
(1)
(R.D)
(R,1)
(D.I)
(R.D.I)
Average
21.38
28.01
34.79
17.19
20.92
21.68
14.39
22.63
PCD
+1.85
+:2.6
+2246
+0.65
+2.41
43.61
#3.54
+2.18
NYUv2
Cityscapes (T)
[R]
(T)
(R,T)
Average
(R)
(R,T)
Average
45.68
44.34
49.44
46.49
78.26
61.30
79.53
73.03
PCD
#0.11
+ 0.08
+ 0.08
+0.04
#0.21
#0.26
#0.29
00.11
Table 7: Ablation study of loss components CASIA-SURF, CeFA, NYUv2 and Cityscapes.
CASIA-SURF
Le
Lu
Cg
{R]
[D]
(1)
(R.D)
(R.I)
[D.I]
(R.D.)
Average
x
x
12.31
2.89
19.24
1.31
8.16
2.19
1.35
x
V
13.55
2.01
18.02
0.86
5.81
2.53
0.85
6,24
x
7.59
4.10
7.97
1.83
3.86
2.04
0.97
4.05
V
v
7.23
2.20
5.66
0.99
2.86
0.89
0.74
2.93
CeFA
C
Lu
[R]
(D)
(1)
[R.D]
(R.I)
(D.I)
(R.D.I)
Average
x
X
26.95
38.06
37.06
24.18
24.75
32.82
25.38
29.89
x
21.14
33.76
37.22
21.28
23.61
27.56
21.19
26.53
X
V
20.62
34.43
35.23
18.18
21.86
32.63
21.72
26.38
21.38
28.01
34.79
17.19
20.92
21.68
14.39
22.63
NYUv2
Cityscapes
Ce
Lu
C
(R]
(T)
(R.T)
Average
(R)
(T)
(R.T)
Average
X
44.24
41.17
47.89
44.43
77.54
59.64
78.46
71.89
X
45.96
42.95
48.54
45.82
78.11
60.62
79.07
72.60
X
44.48
42.02
48.86
45.12
77.52
59.94
78.91
72.17
45.68
44.34
49.44
46.49
78.26
61.30
79.53
73.03
learning rate is initialized to 0.01 and the warm-up epoch is set 150. After the warm-up stage, cosine annealing learning rate strategy is employed.
C Stability Experiments
In Table 6, we detail the stability experiments for PCD across all datasets. Each experiment is repeated for three times to ensure reliability, allowing to calculate the average score along with the standard deviation. The results reveal that, even in its worst-case scenario, PCD outperforms the best competing methods, registering average improvements 0.87% on NYUv2, 0.72% on Cityscapes, 0.76% CASIA-SURF, and a significant 3. 3.13% on CeFA. These outcomes not only underscore PCD's superior performance but also attest to its stability and consistency across wide range of testing conditions. This consistent reliability highlights the robustness and adaptability of PCD, making it an effective solution varied scenarios.
D The Visualization of Feature distribution
use t-SNE visualize the distribution of the modality-complete RGB, Depth, and represen- tations of the unified model without PCD distillation CASIA-SURE, The results shown in
16

---

Figure 5. It can be observed that each unimodal distribution is similar but different to the modality- complete distribution, which provides empirical evidence for PCD to consider the indeterminacy in the mapping from incompleteness to completeness.
(x) Xompley--Comlltte
(b) RGB
(4) Depch
(4) IR
Figure 5: The visualization of the distributions of the modality-complete, RGB, Depth, and IR representations from the unified model without distillation.
E
Ablation Study
E.1
Ablation Study Loss Components
We further conduct ablation studies to evaluate the effects of different loss components the NYUv2, Cityscapes, CASIA-SURE, and CeFA datasets, as presented in Table Notably, incorporating any of the loss components yields substantial improvements, particularly with the CeFA dataset. When applied separately, Lu and Cg each contributed to an average accuracy improvement of 3.36% and 3.51% % respectively. These results underscore the significance of constraining probabilities of extreme points for enhancing the transfer of privileged information. Overall, the PCD model achieves optimal performance when it incorporates all proposed loss components.
E.2 Analysis of Hyperparameter À
To further assess the stability of the PCD model in response various parameters, we report its average performance across
CeFA
25
24
the CASIA-SURF and CeFA datasets, as illustrated in the left panel of Figure 6. The performance curve demonstrates that PCD maintains considerable stability across range of À values, where the performance variance is kept within 0.8. Notably, PCD consistently outperforms SOTA models on all datasets when the À value is between 1.6 and 2. Based on these observations, we have set À to 1.8 throughout tion experiments to ensure optimal perfor- mance and stability. This consistent outper- formance underscores the robustness of the PCD model under varying conditions.
23
ACER
20.9
4.-
35
25
classifica-
14
1.8 7
2.0
02
0.3
0.4
0.5
0.6
Figure 6: The average performance of PCD under different À and 7 values on CASIA-SURF and CeFA.
E.3 Analysis of Hyperparameter
In the right panel of Figure 6, we conducted a series of experiments to evaluate the impact of different values of the hyperparameter on the performance of PCD on the multimodal classification datasets CASIA-SULE -URF and CeFA. This hyperparameter, which acts as the temperature coefficient in Equation (8), is used to scale the similarity measures. The experimental findings indicate that performance of the model is relatively insensitive to variations in within certain range. Based on our results, we chose to set 7 to 0.5 for all subsequent experiments to ensure an optimal balance between performance and parameter sensitivity.
17

---

Table Analysis of warm-up epoch on CeFA.
Epoch
(R]
(D)
[1)
(R.D)
[R.I]
(D.I)
(R.D.I)
Avg
30
19.28
32.35
36.41
17.15
20.26
26.26
16.18
23.98
40
19.29
28.20
36.40
15.30
20.58
24.08
17.71
23.08
50
21.38
28.01
34.79
17.19
20.92
21.68
14.39
22.63
60
21.46
27.16
35.17
16.75
22.97
22.35
15.21
23.01
70
21.64
29.22
35.86
17.90
21.69
23.51
17.16
23.86
80
23.35
26.38
33.83
16.40
25.25
21.14
19.83
23.74
E.4 Analysis of Warm-up
Warm-up stage learns to provide complete modality supervision and a good initialization for the subsequent training process. In this part, investigate the impact of varying warm-up epochs on probabilistic distillation. The experimental results in Table 8 emphasize the importance of judiciously setting the warm-up up epoch. The experimental results show that PCD is not sensitive the number of warm-up up epochs. Within the range of 30 to 80, the average result is around 23%, consistently outperforming the SOTA. We set the number of warm-up epochs to 50 for the classification tasks.
Results on SUN RGB-D Dataset
To further confirm the effectiveness of PCD on segmentation tasks, we conduct experiments on a larger dataset, SUN RGB-D [38]. This dataset has 37 categories of objects and contains 5,285 RGB-Depth pairs for training and 5050 pairs for testing. The results are shown in Table We can see that PCD is effective even on larger segmented dataset.
Table 9: The mIOU(1) of PCD and other methods on SUN RGB-D.
Methods
(R]
(D)
(R.T)
Average
Separate Model
43.94
39.81
47.84
43.86
MMANET
44.73
39.94
47.54
44.07
PCD
45.63
41.43
47.24
44.75.10.22
G Exploration on Modality-Missing Training Data
In Table 10, we conduct experiments on PCD against multiple SOTAs on the scenarios of training data with missing modalities. Specifically, we evaluated the performance on both the CASIA-SURF and CeFA datasets, where each modality of the training data has either 30% or 40% of its data missing. The results clearly indicate that PCD outperforms all other methods both rates. Notably, PCD shows significant performance improvement on CeFA, with gap of 5.39% under the 30% missing modality condition and 10% under the 40% missing modality condition. These results indicate that although PCD is not specifically designed for modality-misiing training data, it is still scalable for this scenario.
H Limitations and Future Explorations
This paper introduces a probabilistic alignment approach between modality- coolpitt-compltte and modality- missing representations to enhance the effective transfer of privileged information. The proposed method is primarily designed for scenarios where all training samples are modality-complete. and misslig occurs exclusively during testing. If modality- data is present during train- ing, knowledge distillation cannot be applied to the modalltyy-isssinn subset of the data. Therefore, in the future, scenarios with missing data during training will be further the focus of our considcration.
18

---

Table 10: Performance under different training data missing modality rates. The best results are in bold and the second-bes, ones are marked with underline. means the performance gap between PCD and the second-bes best results.
CASIA-SURF (ACER !)
Missing
Method
[R]
[D)
(1)
(R.D)
[R.1]
[D.I)
[R.D.1)
Average
MMANET [46]
13.50
3,38
6.57
6.37
3.72
1.83
1.31
4.67
ETMC (14)
7.63
3.62
10.18
1.12
5.21
1.43
0.96
4.31
30%
PCD
8.28
2.13
6.66
1.24
2.66
2.66
0.60
3.18
A
0.65%
1.25%
0.09%
0.12% 1
1.06%
1.23%
0.36%
1.13%
MMANET [46]
14.96
5.22
9.03
3.24
5.14
2.31
2.10
6.00
ETMC [14]
9.38
7.42
7.44
1,41
3.98
3.16
0.58
4.77
40%
PCD
7.14
1.77
10.88
1,08
3.70
1.10
0.88
3.79
A
2.24%
3.45% 4
3.44%
0.33%
0.28%
1.21%
0.30%
0.98%
CeFA (ACER J)
Missing
Method
[R]
[D]
(1)
(R.D)
[R.1]
[D.I)
[R.D.I)
Average
MMANET [46]
28.39
39.61
34.12
34.19
23.39
34.12
27.11
31.56
ETMC [14]
25.96
34.69
38.60
24.15
24.58
31.83
24.03
29.12
30%
PCD
23.42
30.23
34.60
18.34
21.98
24.50
15.07
23.73
A
2.54%
4.46% J.
0.48%
3.81%%
1.41%..
7.33%
8.96%
5.39% 4
MMANET [46]
29.94
43.40
37.29
31,60
28.62
44.97
31,80
35.38
ETMC [14)
24.38
37.82
38.33
25.04
24.39
36.96
24.03
30.13
40%
PCD
24.91
31.23
34.40
21.09
23.98
23.31
16,30
25.03
A
0.53%
6.58%
2.89%
3,95%
0.40%
13.65% 4
7.73%
5.10% 4
I Impact Statements
The method proposed this paper can effectively improve the robustness of the multimodal model. This exploration is of great significance to the real-world inference scenarios that always obtain modality-complete data, such as healthcare and automatic driving. Compared to previous methods, PCD does add lot of parameters and effectively saves computational costs, So far, have not discovered any negative impacts of this method.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction part clearly reflect the main contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation discussed in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
There are no included theoretical results in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides a detailed description of the experimental settings in Section 4 and Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [NA]
Justification:
The code will be released once the paper is published.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaecccccccccccccclt /tweeeeeudeeeeeeeeeiiiiiiii ssssssolicyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The details are provided Section 4 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: For the overall presentation of the paper, do not provide the numerical experiment's 's statistical significance, which is also consistent with concurrent works.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:///nessseeeeeeeeee http:p/////rrpp..pppppppce/e/// httpaa/eeecccsuceelleeeeeeeeee
Answer: [Yes]
Justification:
We have read and strictly followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such safeguard risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: This paper appropriately cites the original paper that produced the code package and considered the dataset in Section 4 and Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "3hwKCdFZ6M", "reviewer": "Reviewer_upEV", "summary": "Summary:\nThis paper studies the robustness under missing modality scenarios. In multi-modal learning, missing modality is a very common problem that might hinder the learning performance of many existing strategies. The authors assume that the modalities’ information redundancy could potentially help the learning under missing modalities. Specifically, for the missing modality, a probability distribution can be formed as an estimation for the real modality value. To leverage such an assumption, two learning properties are proposed, namely extremum property and conformal property. Based on these two properties, the authors estimate the potential modality distribution as a Gaussian distribution, and data points with complete representations are considered positive points, and negative points otherwise. By encouraging the probability extremum objective and geometric consistency objective, the multimodal learning framework can be successfully formulated. Through extensive experiments, the effectiveness of the proposed method is carefully evaluated and justified.", "questions": "Please see the weakness part.", "limitations": "The authors have discussed limitation in the main paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Strengths:\n- This paper studies a very interesting research topic and could have a potential impact in the field of multimodal learning, as well as for realistic application purposes.\n- The proposed salutation is technically solid and novel, which is a good contribution. It would be better if a theoretical framework is proposed to further justify the proposed method.\n- The experiments are extensive and sufficient. Both quantitative comparisons between many recent baseline methods and detailed analysis are provided. Such as ablation study on different modules, analysis of knowledge distillation strategy, hyperparameter sensitivity analysis, and computational overhead. The evaluation is conducted on many well-known dataset, which makes the results convincing.\n- The performance improvement is promising.", "weaknesses": "Weaknesses:\n- There is abuse of notations which are not clearly defined, which makes the reading a bit hard. Moreover, the writing shall be further improved. There is no general logic in formulating the methodology. The authors mostly just plainly demonstrate what is done.\n- A stronger motivation should be proposed to justify the proposed two properties. When introducing the two properties, it is suggested to first identify the key problem in learning with missing modalities. For example, maybe some empirical evidence that some missing modalities still follow a similar data distribution as the complete ones, then it would be reasonable to propose probability extremum property and further design the learning objective. In future versions, addressing this part could further help the quality and readability of this paper.\n- In the experiments, how did the authors control the missing rate of modalities? If there are some modalities in some instances are missing, the missing rate could significantly affect the final learning results. Moreover, the sampling strategy could be essential, how did the authors sample missing modalities? Is it randomly sampled or uniformly sampled?\n- Moreover, what is the difference between learning under datasets with three modalities and two modalities? Does the change of modalities number affect the learning performance of the proposed method? In my opinion, if the modality number changes, the estimation of the probability distribution could change. With more modalities, the estimation could be more accurate. However, if the missing rate also increases due to the introduce of additional modalities, the influence could be unpredictable. Can the authors make some explanation with respect to this part?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time devoted to further comments. We would like to make more detailed explanations to figure out your concerns.\n\n> Q1: Notations are not clearly defined. Moreover, the writing shall be further improved. There is no general logic in formulating the methodology.\n\n**A1**: We really appreciate the reviewer’s detailed review and will comprehensively polish the writing by clarifying the general logic, and carefully proofread the submission to fix the typos and grammatical errors.\n\n> Q2: A stronger motivation should be proposed to justify the proposed two properties. For example, maybe some empirical evidence that some missing modalities still follow a similar data distribution as the complete ones, then it would be reasonable to propose probability extremum property and further design the learning objective.\n\n**A2**: Thanks for your suggestions on our motivation. We use t-SNE to visualize the distribution of the modality-complete, RGB, Depth, and IR representations of the unified model without PCD distillation. The results are shown in Figure 1 in the pdf. It can be observed that each unimodal distribution is similar to the modality-complete distribution, which provides empirical evidence for PCD to consider the indeterminacy in the mapping from incompleteness to completeness.\n\n> Q3: In the experiments, how did the authors control the missing rate of modalities? Moreover, the sampling strategy could be essential, how did the authors sample missing modalities? Is it randomly sampled or uniformly sampled?\n\n**A3**: We are sorry for missing the information about the augment setting. In this paper, we conduct experiments under two modality-missing settings: 1) training with modality-complete data and testing with modality-missing data; 2) training and testing with the modality-missing data. \n\n- Most of the experiments are under setting (1). During training, we **augment each modality-complete sample by simulating all potential missing modality scenarios and randomly sample one of the augmented data as the training sample for the current epoch**. Thus, the missing rate is $\\frac{2^M-2}{2^M-1}$, where $M$ is the number of modalities, and samples with missing modalities are uncertain in each epoch. Furthermore, to investigate the impact of the random augmentation strategy, we conduct additional experiments with extreme augmentation conditions, namely, excluding simulations that only RGB, Depth, or IR modality is available. The results on CASIA-SURF are shown below. It can be seen that when the random augmentation is no longer applied, there is a decrease in performance. These results indicate that **appropriately simulating and sampling various missing scenarios helps enhance the multimodal robustness.** During testing, we build **various testing sets with different missing cases, where each set contains only one missing case using the entire testing data**. We report the results on each testing set, as well as the average results across them. \n\n\n| Augmentations  | {R} | {D} | {I} | {R,D} | {R,I}| {D,I} | {R,D,I} | Average |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| w/o {R} | 7.97 | 2.36 | 10.22 | 1.18 | 4.05 | 1.22 | 0.92  | 3.99 |\n| w/o {D} | 7.39 | 4.48 | 8.59 | 1.53 | 3.55 | 1.42  | 0.65 | 3.94 |\n| w/o {I} | 7.00 | 2.19 | 15.43 | 1.03 | 4.00 | 1.59 | 0.82 | 4.58 |\n| PCD | **6.54** | **1.67** | **8.13** | **0.80** | **2.76** | **0.82** | **0.54** | **3.03** |\n\n- The experiments in Table 5 in the main paper and Table 9 in Appendix are under setting (2). During training, we augment each data by simulating all possible modality-missing cases for this data. During testing, we build the same testing sets as setting (1). In Table 9, we evaluate the performance on both the CASIA-SURF and CeFA datasets, where each modality of the training data has either 30% or 40% of its data missing. As can be seen, **the missing rate for training data could affect the final results under setting (2), where the missing rate is larger, the performance is worse.** This is because that PCD is only applied to the data that has a modality-complete counterpart, a large missing rate means that less data is used for distillation.\n\n\n> Q4: What is the difference between learning under datasets with three modalities and two modalities? Does the change of modalities number affect the learning performance of the proposed method? Can the authors make some explanation with respect to this part?\n\n**A4**: To answer the reviewer's question, in the following, we explore the impact of the number of modalities by controlling it on CASIA-SURF. The experiments are performed under setting (1) and the results are shown below. Notice that, the higher the number of modalities, the larger the missing rate of the training set. For example, the missing rate is 2/3 for two-modality and 6/7 for three-modality. From the results, it can be observed that **with more modalities, better modality-complete representations can be provided, so as to transfer more privileged information.** This results in a model that is relatively robust across various missing cases. **When the number of modalities is small**, there are fewer missing cases that need to be considered. **The model may focus more on fitting an easy missing case, resulting in marginal improvements.** For example, when complete modalities are RGB and Depth, the result of two-modality data at Depth (1.73\\%, lower is better) is better than that of three-modality (2.20\\%).\n\n| Training Modalities | {R} | {D} | {I} | {R,D} | {R,I} | {D,I} | {R,D,I} |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| RGB, Depth | 7.67 | 1.73 | \\ | 1.18 | \\ | \\ | \\ |\n| RGB, IR  | 7.14 | \\ | 14.52 | \\ | 5.61 | \\ | \\ |\n| Depth, IR | \\ | 1.61 |  6.62 | \\ | \\ | 1.54 | \\ |\n| RGB, Depth, IR | 7.23 | 2.20 | 5.66 | 0.99 | 2.86 | 0.89 | 0.74 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aXQ9RyKFbV", "reviewer": "Reviewer_EqMG", "summary": "This paper studied the missing modality robustness problem for multimodal training, by introducing a probabilistic conformal distillation to handle the stringent determinate alignment given the irreparable information asymmetry. Specially, PCD adopts the alignment of the extremum of distribution while maintaining the geometric consistency among modality relation. With extensive experiments, the authors demonstrate the consistent improvement of PCD over the current state-of-the-art methods on several benchmarks.", "questions": "see weakness.", "limitations": "The limiations are discussed in the paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1)\tThe idea of probabilistic conformal distillation is interesting, since the brute-force distillation from complete modality to missing modality is usually ill-posed given the information asymmetry. The authors designed a mild way to simultaneously propagate the task-relevant information and avoid the overfitting. \n\n(2)\tThe instantiation of the probability extremum and geometry consistency for PCD is elegant due to the simplicity and minimal extra cost in implementation. This makes the proposed method easily extended for different missing modality cases as shown in the experimental parts, i.e., segmentation and classification. \n\n(3)\tThe writing is easy to follow and the experiments on the well-used benchmark show the promising improvement over current state-of-the-art methods. A range of the experiments for ablation and further analysis confirm the superiority of PCD.", "weaknesses": "Although the submission is overall good, there are still some minor concerns that need to be validated and addressed, which I summarize as follows.\n\n(1)\tSome equations are misleading. For example, in Eq. (2), it is not clear that whether the authors mean the accumulative probability of z_p^* is larger than z_n^*, or exactly mean each z_p^* to z_n^*. Similar cases happen in Eq. (3). I think the authors should clarify this part due to their intrinsic difference and different implication. Besides, the similarity measure s(\\cdot, \\cdot) cannot be any metric, since it occurs inside the log operation (Eq. (5)) and should be satisfy the positive constraint.\n\n(2)\tIt is not clear why the Eq. (7) and Eq. (9) share the coefficient in Eq. (10). Is this optimization stable? Are they in the similar scale in terms of the loss range. At least, the authors should give some evidences or some ablations to show the rationality about the shared \\lambda in Eq. (10).\n\n(3)\tIt seems that the improvement of the experiments in Table 1 for NYUv2 and Cityscapes are smaller than one. It is better for these experiments with multi-round trials and report the std for the statistical signification explanation. \n\nOverall, I think the idea of probabilistic conformal distillation is an interesting and promising way to combat the brute-force alignment between missing modality and complete modalities during robust training. Carefully improving the submission by considering above weakness will make it more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time devoted to reviewing this paper and your constructive suggestions.\n\n> Q1: Some equations are misleading. For example, in Eq.(2), it is not clear that whether the authors mean the accumulative probability of $z\\_p^\\*$ is larger than $z\\_n^\\*$, or exactly mean each $z\\_p^\\*$ to $z\\_n^\\*$. Similar cases happen in Eq.(3). I think the authors should clarify this part due to their intrinsic difference and different implication. Besides, the similarity measure $s(\\cdot, \\cdot)$ cannot be any metric, since it occurs inside the log operation (Eq.(5)) and should be satisfy the positive constraint.\n\n**A1:** Eq.(2) requires that the probability of any one positive point $z\\_p^\\*$ is larger than the probabilities of all negative points $z\\_n^\\*\\in Z\\_n$. Eq.(3) has a similar meaning, namely, for any $g^⋆\\_p \\in G\\_p$, the inequality $s(g^⋆\\_p, g\\_i) \\gg s(g^⋆\\_n \\in G\\_n, g\\_i)$ should be hold. And for $s(\\cdot, \\cdot)$, thanks for pointing out the areas where our presentation is unclear.\n\n> Q2: It is not clear why the Eq. (7) and Eq. (9) share the coefficient in Eq. (10). Is this optimization stable? Are they in the similar scale in terms of the loss range. At least, the authors should give some evidences or some ablations to show the rationality about the shared $\\lambda$ in Eq. (10).\n\n**A2:** The Eq.(7) and Eq.(9) are realizations of the Probability Extremum and Geometric Consistency components of Eq(5), respectively. And Eq.(5) is the simplification of the objective Eq.(4). **In this simplification process, no distinct coefficients are generated for the two terms in Eq.(5).** Thus we share the coefficients of Eq.(7) and Eq.(9) in Eq.(10). Besides, we also conduct experiments on CeFA to explore the impact of using different coefficients. The results are shown below. As can be seen, setting different hyperparameters $\\lambda\\_{u},\\lambda\\_{g}$ does not cause significant performance fluctuations.\n\n| $\\lambda\\_{u}$  | 1.4 | 1.6 | 1.8 | 2.0 | 2.2 | Best Baseline |\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n| Avg ACER $(\\downarrow)$ | 23.36 | 24.69 | 22.63 | 23.85 | 22.56 | 27.94 |\n| $\\lambda\\_{g}$ | **1.4** | **1.6** | **1.8** | **2.0** | **2.2** | **Best Baseline** |\n| Avg ACER $(\\downarrow)$ | 22.10 | 22.99 | 22.63 | 23.41 |22.15  | 27.94 |\n\n> Q3: It seems that the improvement of the experiments in Table 1 for NYUv2 and Cityscapes are smaller than one. It is better for these experiments with multi-round trials and report the std for the statistical signification explanation.\n\n**A3:** **In Table 6 in Appendix**, we detail the stability experiments for PCD across four datasets. Each experiment is repeated for three times, allowing to calculate the average score along with the standard deviation. **The results reveal that, even in its worst-case scenario, PCD outperforms the best competing methods.** These outcomes not only underscore PCD’s superior performance but also attest to its stability and consistency across a wide range of segmentation testing conditions. In addition, to further confirm the effectiveness of PCD on segmentation tasks, we conduct experiments on a larger dataset, SUN RGB-D[1], which contains 5,285 RGB-Depth pairs for training and 5050 pairs for testing. The results are shown below. We can see that PCD is effective even on a larger segmented dataset.\n\n| Methods | {R} | {D} |{R,T} | Avg |\n|:-:|:-:|:-:|:-:|:-:|\n| Separate Model | 43.94 |  39.81 | 47.84 | 43.86 |\n| MMANET | 44.73 | 39.94 | 47.54 | 44.07 |\n| PCD | $45.63\\_{\\pm0.16}$ | $41.43\\_{\\pm0.07}$ | $47.24\\_{\\pm0.17}$ | $44.75\\_{\\pm0.02}$ |\n| $\\Delta$ | 0.90 | 1.49 | -0.30 | 0.68 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P5jau4uAGm", "reviewer": "Reviewer_UGMA", "summary": "By formulating the missing modality representation as a probability distribution, this work proposes probabilistic conformal distillation (PCD), which is an objective function that encourages 1. consistent latent representation of multimodal embeddings for data points in the same class. 2. Geometric consistency of inter-class latent representation. Simulations shows the empirical success of such strategy in improving the missing modal robustness for student models.", "questions": "See weakness", "limitations": "No", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The overall structure of the paper is well organized and easy to follow.\n2. The proposed loss function (PCD) intuitively makes sense.\n3. Simulations show a clear advantage for the proposed strategy in classification.", "weaknesses": "1. My major concern about PCD lies at its potential use cases. The reasons are two folds. First, the whole framework relies on a definition of positive and negative points, which is not obvious for a lot of multi-modal learning problems (For example, cross-modal retrieval (without class labels), missing-modal imputation, etc). At this point, the paper only successfully demonstrated its advantage in classification (I will talk about segmentation later). Second, the scalability of the proposed approach is also of a minor concern, and the main reason is due to the loss L_g. As this loss seems to scale quadratically with batch size, the time and memory complexity of PCD needs to be formally analyzed.\n2. For segmentation, the improvements seem marginal at best, therefore, it is important to provide error bars to validate whether there is any statistically significant difference. \n3. Still for segmentation, it is mentioned in line 134 that the positive group for each sample only contains itself. In this case, L_u, if I understand correctly, just encourages the latent representation of all samples to be distinctive from each other, and I do not see the purpose of L_g in this case, as there is nothing special about the inter-class geometry at this point. It would be important for the author to conduct a simulation similar to Table 2, but for segmentation, to see whether the inclusion of these losses make any difference (Sepecially given that the segmentation simulations only show marginal improvments).\n4. There are some writing issues that need to be addressed. \nFor example, equation (1) is about the modal of a distribution, while it is described as probabilistic peak expectation in line 118. Please note that there is a rich literature in modal regression, and these two concepts should not be confused. \nThere is also a value inconsistency in Table 1. Notice in the first column of CASIA-SURF results, the difference between ETMC and PCD is (7.91-7.23 = 0.68), while the improvements in marked as 0.74.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: First, the whole framework relies on a definition of positive and negative points, which is not obvious for a lot of multi-modal learning problems (For example, cross-modal retrieval, missing-modal imputation, etc). Second, as the loss $L\\_g$ seems to scale quadratically with batch size, the time and memory complexity of PCD need to be formally analyzed.\n\n- We are sorry that there may be somewhat misunderstanding about the mission of PCD. **PCD focuses on robust multimodal fusion and requires that the maximum number of input modalities be greater than 1.** Therefore, **mainstream cross-modal retrieval and generation tasks, which typically have only a single input modality**, are not suitable for the missing-modality problem explored by PCD. As for the recent popularity of the multimodal retrieval task, where the query text prompts modifications in the query image, will become invalid if any input modality is missing. Hence, it does not align with the problem that PCD aims to solve. Besides, although PCD relies on the definition of positive and negative points, **class labels are not necessary.** When class labels are available (e.g. the classification tasks), modality-complete representations sharing the same class with the modality-missing input are considered positive points, while the ones from different classes are negative points. When class labels are unavailable (e.g. the segmentation tasks), only the modality-complete representation of the same sample is considered positive, and all others are considered negative, as we have set up in the segmentation task.\n-  To address the concern about computational cost, we estimated the memory complexity and the time for each iteration of PCD and the other three methods with the same batchsize (64) on CeFA. It can be seen that PCD does not significantly increase training time and memory complexity.\n\n|Method|MD|RAML|MMANet|PCD|\n|:-:|:-:|:-:|:-:|:-:|\n|Memory (G) |4.371|3.285| 3.621|3.809|\n|Time per iteration (s)|0.155|0.104|0.106|0.174|\n\nIn addition, we explore the relationship between batch size and memory complexity, and computation time. **The results indicate that a larger batchsize is not always better.** The optimal batchsize (64) has a relatively low computational cost.\n\n| Batch Size | 32 | 64 | 128 | 192 |\n| :-: | :-: | :-: | :-: | :-: |\n| Memory (G) | 3.145 | 3.809 | 5.739 | 8.081 |\n| Time per iteration (s) | 0.151 | 0.174 | 0.195 | 0.268 |\n| Avg ACER ($\\downarrow$) | 22.70 | 22.63 | 23.79 | 25.16 |\n\n> Q2: For segmentation, the improvements seem marginal at best, therefore, it is important to provide error bars to validate whether there is any statistically significant difference.\n\n**In Table 6 in Appendix**, we detail the stability experiments for PCD across four datasets. Each experiment is repeated for three times, allowing to calculate the average score along with the standard deviation. **The results reveal that, even in its worst-case scenario, PCD outperforms the best competing methods.** These outcomes not only underscore PCD’s superior performance but also attest to its stability and consistency across a wide range of segmentation testing conditions. In addition, to further confirm the effectiveness of PCD on segmentation tasks, we conduct experiments on a larger dataset, SUN RGB-D[1], which contains 5,285 RGB-Depth pairs for training and 5050 pairs for testing. The results are shown below. We can see that PCD is effective even on a larger segmented dataset.\n\n| Methods |{R}| {D}|{R,T}|Avg|\n|:-:|:-:|:-:|:-:|:-:|\n|Separate Model|43.94|39.81|47.84|43.86|\n|MMANET |44.73|39.94|47.54|44.07|\n|PCD | $45.63\\_{\\pm0.16}$ | $41.43\\_{\\pm0.07}$ | $47.24\\_{\\pm0.17}$ | $44.75\\_{\\pm0.02}$ |\n| $\\Delta$|0.90|1.49 | -0.30 | 0.68 |\n\n> Q3: For segmentation, Lu, just encourages the latent representation of all samples to be distinctive from each other, and I do not see the purpose of Lg, as there is nothing special about the inter-class geometry. It would be important to conduct ablation studies for segmentation, to see whether the inclusion of these losses make any difference.\n \nWe are sorry for the misunderstanding caused by the lack of clarity. \nPCD aims to model different modality-missing representations as distinct distributions to fit their unknown PDFs. This is realized by considering properties of positive and negative points on the modeled distribution $q(z|\\mathrm{x}\\_i)$. It is important to note that **all positive and negative points are generated by a pretrained modality-complete model and remain fixed during training.** In a segmentation task, for a modality-missing input, there is **only one positive point $z_p^\\*$ for the PDF** of the input's mapping variables in the modality-complete space, namely, **its corresponding modality-complete representation**. Here, the **negative points $z_n^\\*$ are modality-complete representations of all other samples**. The loss **$L_u$** is used to maximize the probabilities of $q(z|\\mathrm{x}_i)$ at $z_p^\\*$ and minimize them at $z_n^\\*$. This **implicitly encourages the mean of $q(z|\\mathrm{x}\\_i)$ closer to $z\\_p^\\*$ and further away from $z\\_n^\\*$**. Besides, in **$L\\_g$**, the structures are represented by calculating the distances of peak points of $q(z|\\mathrm{x}\\_i)$ and $z^⋆$, respectively, where **$z^⋆$ is no longer divided into $z\\_p^\\*$ and $z\\_n^\\*$**. The complete ablation studies is shown in **Table 7 in Apendeix or Table 1 in the pdf (the same)**, which verify the effectiveness of $L\\_u,L\\_g$.\n \n> Q4: There are some writing issues that need to be addressed. Equation (1) is about the modal of a distribution, while it is described as probabilistic peak expectation in line 118. There is also a value inconsistency in Table 1.\n\nWe really appreciate the reviewer’s detailed review and will carefully proofread the whole submission to fix the imprecise expressions and typos. Specifically, in the revision, we will modify line 118 to be the peak of probability and recalculate the Table 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We gratefully thank all the reviewers for their devoted efforts and constructive suggestions on this paper. We are glad that the reviewers have some positive impressions of our work, including: 
- The overall structure of the paper is **well-organized and easy to follow.** (Reviewer EqMG, UGMA)
- The explored research topic is **interesting** and **could have a potential impact in multimodal learning and realistic application purposes**. (Reviewer upEV, EqMG)
- The method is **novel, technically solid**, and **can easily be extended to different missing modality task**. (Reviewer upEV, EqMG)
- **Extensive, sufficient and rigorous** experiments with **comprehensive ablation study and analysis** (Reviewer upEV, EqMG, UGMA). 

We have addressed the reviewers' comments and concerns in **individual responses to each reviewer**. The reviews allowed us to improve our draft, and the changes made in our responses are summarized below:

- We conduct several experiments to analyze the impact of the missing rate, the sample strategy, and the number of modalities. (A3, A4 to Review upEV).
- We explore the sensitivity of using different hyperparameters $L_u, L_g$ to optimize the loss function Eq.(10) (A2 to Review EqMG).
- We showcase the stability of PCD in four datasets. (A3 to Review EqMG, A2 to Review UGMA).
- To address the reviews concern about computational cost, we compared PCD with other SOTA methods in terms of memory complexity and the time for each iteration (A1 to Review UGMA).
- We provide further clarifications on several points, including the empirical evidence about motivation (A2 to Review  upEV), the details of equations (A1 to Review EqMG), the application scope of the algorithm (A1 to Review UGMA) and the effectiveness of two loss terms (A3 to Review UGMA).


**We appreciate all reviewers' great effort again!** We are looking forward to your reply.

Below is the list of the references that can be used in the responses.

[1] Song S, Lichtenberg S P, Xiao J. Sun rgb-d: A rgb-d scene understanding benchmark suite[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 567-576.

Meta-review: The paper addresses the problem of missing modalities in multi-modal learning. The authors propose leveraging the information redundancy across modalities to improve learning under missing modalities.

Specifically, the authors assume that for a missing modality, a probability distribution can be estimated to approximate the real modality value. They introduce two key properties - the extremum property and the conformal property - to guide the estimation of this probability distribution, modeled as a Gaussian in the paper.

The two properties are considered in two objective loss functions as follows:

A. An "extremum probability" objective, which says that the complete modality value in latent representation occurs at the maxima of the missing modality probability distribution;
B. A "geometric consistency" objective, which maintains the geometric relationship between the latent representations of missing modalities and the complete modality within a batch of samples, exploiting a contrastive learning loss.

Experimental evaluation is based on several benchmarks in image classification and segmentation, which shows that this "Probabilistic Conformal Distillation" (PCD) method consistently outperforms state-of-the-art approaches in improving robustness to missing modalities.

In summary, the key contributions are the probabilistic modeling of missing modalities and the novel PCD learning objective that leverages extremum and conformal properties to enhance multimodal learning performance in the presence of missing data. Experimental results are encouraging. 

Concerns from the reviewers are mostly from the clarity and preciseness in elaborations of the methodology, as well as the application scenarios. Among the various notational and clarity issues raised by reviewers, a particular one is about the usage of contrastive learning loss $L_g$ in geometric consistency objective. The confusion mostly comes from that the paper is introduced mainly through the perspective of classification where the positive samples arise from the same class (e.g. Eq. (7-9) as definitions) while in segmentation positive sample is just one which is only discussed briefly in line 134. However, in practice, positive samples in contrastive learning may be also generated from various data augmentation techniques from the same image. In the case of a single positive sample, $L_g$ is probably a stronger objective than extreme objective as it imposes more geometric relationship on different samples within the batch. 

Although the confusions exist in those parts about notations (e.g. $\alpha$ is a distance but inner product is not) and elaborations, such concerns could be clarified in the final version without too much new studies. Therefore the current manuscript could be accepted if the authors make all the clarifications raised by the reviewers carefully.
Decision: Accept (poster)