Title: Generalized Multimodal Fusion via Poisson-Nernst-Planck Equation
Venue: NeurIPS 2024
Accepted: no
Keywords: Multimodal Fusion

Poisson-Nernst-Planck Equation

Feature Disentanglement

Downstream Task Adaptation
Abstract: Previous studies have highlighted significant advancements in multimodal fusion.    Nevertheless, such methods often encounter challenges regarding the efficacy of feature extraction, data integrity, consistency of feature dimensions, and adaptability across various downstream tasks.
This paper proposes a generalized multimodal fusion method (GMF) via the Poisson-Nernst-Planck (PNP) equation, which adeptly addresses the aforementioned issues.
Theoretically, the optimization objective for traditional multimodal tasks is formulated and redefined by integrating information entropy and the flow of gradient backward step.      Leveraging these theoretical insights, the PNP equation is applied to feature fusion, rethinking multimodal features through the framework of charged particles in physics and controlling their movement through dissociation, concentration, and reconstruction.
Building on these theoretical foundations, GMF disassociated features which extracted by the unimodal feature extractor into modality-specific and modality-invariant subspaces, thereby reducing mutual information and subsequently lowering the entropy of downstream tasks. 
The identifiability of the feature's origin enables our approach to function independently as a frontend, seamlessly integrated with a simple concatenation backend, or serve as a prerequisite for other modules.
Experimental results on multiple downstream tasks show that the proposed GMF achieves performance close to the state-of-the-art (SOTA) accuracy while utilizing fewer parameters and computational resources. Furthermore, by integrating GMF with advanced fusion methods, we surpass the SOTA results.

Abstract

Previous studies have highlighted significant advancements in multimodal fusion. Nevertheless, such methods often encounter challenges regarding the efficacy of feature extraction, data integrity, consistency of feature dimensions, and adaptability across various downstream tasks. This paper proposes a generalized multimodal fusion method (GMF) via the Poisson-Nernst-Planck (PNP) equation, which adeptly addresses the aforementioned issues. Theoretically, the optimization objective for traditional multimodal tasks is formulated and redefined by integrating information entropy and the flow of gradient backward step. Leveraging these theoretical insights, the PNP equation is applied to feature fusion, rethinking multimodal features through the framework of charged particles in physics and controlling their movement through dissociation, concentration, and reconstruction. Building on these theoretical foundations, GMF disassociated features which extracted by the unimodal feature extractor into modality-specific and modality-invariant subspaces, thereby reducing mutual information and subsequently lowering the entropy of downstream tasks. The identifiability of the feature's origin enables our approach to function independently as a frontend, seamlessly integrated with a simple concatenation backend, or serve as a prerequisite for other modules. Experimental results on multiple downstream tasks show that the proposed GMF achieves performance close to the state-of-the-art (SOTA) accuracy while utilizing fewer parameters and computational resources. Furthermore, by integrating GMF with advanced fusion methods, we surpass the SOTA results.

1 Introduction

The world is inherently multimodal; individuals perceive and integrate diverse sensory inputs to form a more comprehensive understanding of their surroundings. Similarly, multimodal learning processes inputs from multiple modalities, offering potential applications in complex downstream tasks such as cross-modal retrieval and multi-modal classification. Nevertheless, features from different modalities often differ significantly, even when describing the same event [1, 2]. Consequently, fusing features from different modalities is challenging, requiring a dedicated fusion phase before being applied in tasks, bridging the semantic gap between different modalities is crucial for valid feature fusion.

Theoretical works on multimodal fusion have proposed more generalized schemes. MBT [3] ex30 changes mutual information between different modalities to enhance understanding. Perceiver [4] 31 stacks various features and extracts fusion features from transformer blocks to condense task-related 32 features. Uni-Code [2] distinguishes between modality-invariant and modality-specific features, 33 optimizing feature utilization. Moreover, in downstream tasks, innovative fusion methods are applied. 34

MAP-IVR [5] considered that image features belong to the subset of video features, UA VM [6] fuses 35

different modalities using an independent fusion block. 36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

Although existing methods for feature fusion show considerable improvements, they often rely on several incomplete assumptions: 1)Feature dimension consistency: Feature dimensions across different modalities are perfectly aligned [7, 8], leading to inefficient representations, thus impairing model performance; 2)Data reliability: In reality, poor quality data (e.g. missing modalities) directly degrades performance [9, 10], even though datasets are assumed to be complete; 3)Downstream task applicability: Feature fusion requirements are uniform across different tasks, but matching tasks [11, 12, 13, 14, 5] require modality-invariant features (common to all modalities), whereas detection tasks [15, 16] necessitate modality-specific features (specific to each modality) additionally; 4)Feature extraction effectiveness: Loss function in feature fusion does not affect the feature extractor's gradients [17, 18] (See Appendix A), often results in feature extractor homogenization [17], deteriorating performance in downstream tasks [1]. Furthermore, the fixed quantity of modal features often limit the generalizability of proposed fusion methods [2].

This paper introduces a generalized multimodal fusion method (GMF) that operates independently of the usual constraints. We formulate the learning objectives for traditional multimodal tasks and propose new definitions based on information entropy theory [19, 20]. Taking inspiration from the Poisson-Nernst-Planck equation (PNP) [21], treating features as charged particles to disassociate them, employing GMF for multimodal feature fusion. Leveraging the principles of the PNP equation, GMF orchestrates the guided migration of features within a high-dimensional space, segregating modalityinvariant from modality-specific features within the disassociated feature landscape, reducing the mutual information between features further decreases the relevant entropy of downstream tasks. Specifically, the proposed method incorporates a reversible feature dissociation-concentration step and applies reasonable regional constraints to the reconstruction gradient, emphasizing the connection between the feature extractor and the loss of a downstream task, enabling GMF to generalize effectively and serve as the frontend for other fusion modules. We evaluated our method on multiple datasets across specific downstream tasks. It consistently demonstrated significant performance and generalization capabilities. In summary, our contributions are as follows:

We propose a novel theory for multimodal feature fusion based on the Poisson-Nernst-Planck equation and information entropy with an exhaustive proof, demonstrating its effectiveness through theoretical analysis and preliminary experiments.

We have devised a generalized feature fusion method GMF, grounded in entropy theory and the PNP equation, which stands independent of both feature extractors and downstream tasks.

Experiments demonstrate that GMF achieves comparable performance to SOTA with fewer computational demands and parameters, while also showing robustness to missing modalities. Moreover, when integrated with advanced fusion methods, its performance and robustness are notably enhanced, surpassing SOTA and ensuring greater reliability in real-world applications.

2 Related Works

Innovative advancements in multimodal fusion methods, both theoretically [2] and structurally [4], have significantly propelled the progress of generalized multimodal tasks (denote as GMTs ). Some SOTA methods focusing on downstream tasks propose fusion methods specifically tailored for them. However, the fusion challenges vary with the diversity of downstream tasks. In this paper, we categorize multimodal tasks into two types: Native Multimodal Tasks (denote as NMTs ) and Extended Multimodal Tasks (denote as EMTs ), based on whether corresponding single-modal tasks exist. Specifically, cross-modal retrieval and matching tasks such as Image-Video retrieval [14, 5] and Image-Text matching [12, 13, 11] usually belong to NMT and only require the similarity of modalities. For example, CLIP [22] transforms the image classification task into an image-text retrieval task, achieving stunning zero-shot performance. Multi-modal classification, recognition, and detection tasks such as emotion recognition [16] and event classification [6] usually belong to EMT. Different modalities often have inconsistent perspectives, and fully aligned features will affect the performance of such tasks.

modalities during inference. Lastly, "Complexity" reflects computational complexity regarding ( n ) 90 modalities. Perceiver [4] does not report multimodal correlation experiments. 91

It is worth noting that the evaluation of gradient correlation is simply whether there is an explicit 92 excitation of the loss function. Some downstream methods introduce ways such as concat (e.g., 93 classifier of A V oiD-DF [15]) in the classification stage, and the modal missing adaptation in the fusion 94 stage does not represent the adaptation for this task. In addition, for NMTs, the complete modal input 95 is necessary, so the conclusion of this part is "-"; Here, the complexity takes the highest value, which 96 does not represent the final computation cost. (e.g., the disentangled loss of MISA [16] is O ( n 2 ) . 97

98

3 Theory

In this subsection, we briefly introduce the notation system used in this paper and the general structure 99 of multimodal tasks, representing the information entropy at different stages of multimodal learning. 100 After that, we generalize the information entropy to multi-modality and redefine the entropy reduction 101 objective for multi-modal learning. Finally, we evaluate the impact of linear dimension mapping on 102 the performance of downstream tasks and present the preamble theorem.

ΓςΓOΓDΓVΓVΓLΓIΓLΓFΓDΓWΓLΓRΓQ

Γ↼Γ♭ΓWΓUΓDΓFΓWΓLΓRΓQ

Γ↽ΓXΓVΓLΓRΓQ

3.1 Formulation and Traditional Objective Definition

Consider inputs with d modalities, where j ∈ { 1 , 2 , . . . , d } represents different modalities. Examine a dataset comprising n samples. Let the input be X = { X 1 , X 2 , . . . , X n } , where a specific sample i ∈ { 1 , 2 , . . . , n } is represented as X i = { X (1) i , X (2) i , . . . , X ( d ) i } . The output is Y = { Y 1 , Y 2 , . . . , Y n } , and each { X i , Y i } forms a sample pair. X ( j ) i represents the original sample of modality j with varying shapes, while the shape of Y i depends on the specific datasets and downstream tasks. For each modality j , specific feature extractors f ( j ) ( · , θ ( j ) ) and parameters θ ( j ) are employed for feature extraction. The fused features capturing multimodal interactions for sample i are denoted as Z i = { Z (1) i , Z (2) i , · · · , Z ( d ) i } . The set of global fea-

103

104

105

106

107

108

109

110

111

112

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

The multimodal task is depicted in Figure 1, delineating three key parameters: the feature extractor θ , fusion parameter θ F , and classifier parameter θ C . Optimization of these parameters aims at maximizing performance. Regarding entropy, F ( · ) represents the fused mapping, extending the learning objective from feature extraction to fusion:

Similarly, we employ C ( · ) to represent the mapping for downstream tasks and generalize it to embody the learning objective fused with downstream tasks:

In Eq. ( 2), these parameters are optimized by downstream task losses. If there is a loss in the fusion stage, then it optimizes the parameters in Eq. ( 1).

3.2 Information Entropy and Objective Redefinition

Feature extraction through dimensionality reduction involves reducing data uncertainty [19], as quantified by information entropy H . In Figure 1, we show a simplified approach to single-modal learning. The feature extractor and classifier (dotted arrow) directly minimize the information entropy of both the input X ( j ) i and the output Y i by adjusting the parameters of the feature extractor f ( j ) ( · , θ ( j ) ) and the classifier C ( j ) ( · , θ C ( j ) ) for modality j :

This process, facilitated by feature extractors, condenses data samples into a feature space, preserving pertinent attributes for downstream tasks. Think loss as stimulation of entropy reduction, maximize mutual information about related features [18]. Expanding to the multimodal fusion stage, the objective is to minimize the entropy of the fused features compared to the sum of the entropy of each input feature. In the context of multimodal fusion, where outputs from disparate modalities are integrated post-feature extraction, the total information entropy of the system can be estimated using the joint entropy formula, and for constant X :

Downstream objectives are typically structured to minimize mutual information, consequently leading 136 to a reduction in entropy. However, in fusion stage, disparities observed among the equations (1), 137 (2), and (3) suggest that certain fusion-method might not establish a straightforward correspondence 138 between network inputs and outputs. Achieving complete consistency between modalities, where 139 mutual information is zero, may not always lead to optimal outcomes [1, 20], potentially increasing 140 entropy in downstream task-related features [17]. This observation is substantiated by the diminishing 141 performance of certain multimodal methods [3, 15] compared to earlier unimodal methods, indicating 142 a decline in their capacity to extract distinctive features from individual modalities when confronted 143 with the absence of certain modalities. Thus, optimization objectives for multimodal tasks should 144 balance minimizing entropy during fusion with maintaining or reducing entropy in downstream 145 task-related features. This highlights the necessity of aligning deep learning tasks with downstream 146 objectives and minimizing information entropy when designing loss functions for these tasks. 147

Theorem 3.1: The overarching objective of multimodal tasks lies in minimizing entropy during the 148 fusion stage without amplifying the entropy of downstream task-related features: 149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

Some approaches introduce the fused results as residuals, which demonstrate a certain degree of improvement, and this theory provides a better rationale for such enhancement. However, given that the forward pass necessarily involves the operation of F ( · ) , it becomes challenging to fully meet this precondition. During gradient backward, the loss incurred during the fusion stage for the feature extractor should align with the loss of the downstream task or be zero.

3.3 Modality Feature Dissolution and Concentration

Adding too many parameters, or overcharacterization, can improve the model's ability to fit the data, acting like a parameterized memory function [23]. However, it's important to balance this with the amount of data available for the next task to prevent learning too much noise and overfitting [7]. On the other hand, having too few parameters may weaken the model's ability to represent complex patterns, resulting in lower performance across different methods (See Appendix C).

Theorem 3.2: The dimension of the feature that is best suited to the downstream task varies, and there is always an optimal value for this feature. The dimension multiple relationship between each layer of the feature extractor is fixed, and the initial dimension is adjusted. Too low dimension of the final output will lead to inefficient representation, and too high dimension will introduce noise. The existence of an integer l best such that for any integer l distinct from l best, the conditional entropy of the model's predictions f l ( X,θ l ) is greater than that of the model's predictions f l best ( X,θ l best ) .

̸

Theorem 3.3: The feature extractor is fixed, and its original output feature dimension l is mapped to nl , and finally back to l . The mapping result is used as the basis for the downstream task. The performance of downstream tasks is infinitely close to the original performance as n increases, but never greater than the original performance. For magnification n > 1 , n ∈ Z , mapping matrix U 1 ∈ R l × nl and U 2 ∈ R nl × l , For the output features f ( X,θ ) ∈ R l and Y :

Conjecture 3.1: Rely on Theorem 3.1, 3.2, 3.3, we propose an conjecture that a boundary of performance limitation exists, determined by downstream-related entropy. Theoretically, by establishing a direct correspondence between the extractor and classifier, fusion method can enhance the limitation boundary, further improve performance.

3.4 Poisson-Nernst-Planck Equation

The Nernst-Planck equation represents a mass conservation equation that characterizes the dynamics of charged particles within a fluid medium. This equation modifies Fick's law of diffusion to include scenarios where particles are also mobilized by electrostatic forces relative to the fluid. The equation accounts for the total flux of particle p ∈ { + , -} , denoted as J p , of charged particles, encompassing both diffusion driven by concentration gradients and migration induced by electric fields. Since fusion features are usually one-dimensional, we only consider the x direction here. For a given charged particle i , the equation describes its movement as follows:

p is abstracted as elements in the modality-invariant feature and the modality-specific feature. Here, 185 c p ( x, t ) denotes the concentration of particle, while D p (diffusivity of p ), k B (Boltzmann constant), 186 z p (valence also electric charge), and e (elementary charge) are constants. T is a hyperparameter, 187 represent temperature. E represents the electric field of the entire system, and v represents the flow 188 rate. The Poisson equation describes the relationship between the distribution of a field and the 189 potential energy it induces, represented by the expression: 190

ϕ signifies the potential, considered as an external excitation, ε 0 represent dielectric constant. By 191 integrating the relationship between the concentration of charged particles and the electromigration 192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

term in the Poisson equation, we derive the Poisson-Nernst-Planck (PNP) equation. Assuming that the dissociation process approaches equilibrium, for feature elements without magnetic field and flow velocity, we can consider the time-dependent change in concentration c p ( x, t ) of the charged particle i over time t is negligible:

When the final state is stable, a sufficiently large 1D electrolytic cell of length l , at the potential equilibrium boundary b , it can be equivalent to (See Appendix B):

In this context, ϕ ( x ) represents an external influence from another modality feature. We assume that modality-invariant feature elements have a positive charge, while modality-specific feature elements have a negative charge. The difference ϕ (0) -ϕ ( b ) indicates the enrichment potential of modalityinvariant feature elements for the excitation modality. This potential attracts modality-specific feature elements in dissociated modality towards dissociation.

Theorem 3.4: Following dissociation and Theorem3.3, in line with the principles of matter and information conservation, the excitation and attraction features can revert back to their original state. A cyclic feature electrolytic cell is generalized, using a loss function as stimulation:

L is loss function. l ( j ) and b ( j ) are feature dimension and dissociation boundary of modality j , respectively. Around this boundary, features are explicitly distinguished. The mapping matrix U ( j ) dis ∈ R nl ( j ) × l ( j ) , U ( j ) con ∈ R l ( j ) × ( nl ( j +1) + b ( j ) -b ( j +1) ) is learnable. ˆ Z ( j ) i ∈ R nl ( j ) is the result of f ( j ) ( X ( j ) i , θ ( j ) ) ∈ R l ( j ) being linearly mapped (dissolved) into a higher dimensional space.

4 Methodology

Set the dissociation boundary b ( j ) and feature dimension l ( j ) of modality j . The feature with the smallest dimension is denoted as l ∗ . The feature dimension of the dissociation is nl ( j ) , with a uniform magnification of n > 2 .

Combining information entropy theory with the PNP equation, we propose GMF method to optimize fusion feature mutual information on the premise of maintaining the downstream task related information of input features. Following Assumption3.1, GMF has only four learnable matrices for each modality, enforces correlations without complex structure, as shown in Fig 2.

GMF is divided into three stages, for each modality j , applying different learnable mapping matrices: dissolve matrix P ( j ) dis ∈ R nl ( j ) × l ( j ) , concentrate matrix P ( j ) cinv ∈ R b ( j ) × l ∗ and P ( j ) cspec ∈ R ( nl ( j ) -b ( j ) ) × l ( j ) , reconstruct matrix P ( j ) recon ∈ R l ( j ) × ( l ( j ) + l ∗ ) .

First, to make sure the features move, we map (dissolve) them to higher dimensions. Next, for the feature of each modality, after dimension elevation, the goal is explicitly divided as specific and invariant by abstracting different kinds of features into positive and negative charged particles:

f ( j ) ( X ( j ) i , θ ( j ) ) ∈ R l ( j ) , and ˆ Z ( j ) i ∈ R nl ( j ) . Referencing Eq.( 4), irrespective of the initial length l ( j ) of a feature, partitioning it into invariant ( Z ( j ) i ) inv ∈ R l ∗ and specific ( Z ( j ) i ) spec ∈ R l ( j ) components aims to minimize output feature dimensions, thereby mitigating entropy disturbance.

After concentrate, finally, the output Z ( j ) i ∈ R ( l ( j ) + l ∗ ) is obtained:

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

Multi-Modal

input

Multi-Modal

Sorted output

－

+

+

－

Concentration

Disassociated

Modality A

Modality B

recon A

output A

Dissociation

Material

conservation

Loss

Recon-

sturction

Reconstruction

specific

invariant

Diluted Feature

linear

target

Original Feature

Recon-

sturction

.

Eventually the entire system can be restored to its original state. A loss function is given as an 230 external incentive to force the features to move in different directions. Following the Theorem3.4, we 231 use P ( j ) recon to map the features back to f ( j ) ( X ( j ) i , θ ( j ) ) and apply the disassociation loss. 232

5 Experiment

In this section we briefly introduce the experimental dataset, evaluation metrics, implementation details, experimental results and analysis. Our evaluation focuses on solving the limitations mentioned in Section 1 and verifying our theory and hypothesis, so we pay more attention to the fusion performance under the same feature extraction ability.

5.1 Datasets and experimental tasks

We performed the NMT task for image-video retrieval on ActivityNet [24] dataset and the EMT task for audio-video event classification on VGGSound [25] and deepfake detection on FakeAVCeleb [26], and compared the NMT, EMT and GMT methods (as defined in the Related Work) respectively. We conduct three sets of comparison experiments:

Input the same features to simulate the freezing of the feature extractor, and evaluate the entropy reduction effect of the fusion method on the existing information.

Complete the training of the whole model including the same feature extractor, and evaluate the impact of the fusion method on the gradient of the feature extractor.

Select a set of method-specific feature extractors to test the limitation performance.

For EMTs, VGGSound dataset evaluate (1) and (2) 1 , the evaluation metric is the classification accuracy ACC(%). FakeAVCeleb dataset evaluate (3), due to the imbalance of data samples, the evaluation focuses on the Area under the Curve of ROC (AUC). For NMTs, ActivityNet dataset evaluate (4), the evaluation metric is the matching accuracy mAP, mAP@ n represents that the matching task target is selected from n samples.

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

5.2 Implement details

The baseline of the multi-modal is all the direct connection of the features of the output of the single-modal baseline. GMF stands for simple connection as the back-end. "G-method name" stands for GMF as a front-end for the method, See Figure 13 in Appendix for the detailed structure.

5.3 Evaluation

For EMTs, our experiments, detailed in Table 2 and conducted on the VGGSound dataset [25], employ R(2+1)D [28] as the video feature extractor and ResNet-18 [29] as the audio feature extractor. The 'Training Extractor' label indicates trainable parameters, while 'Frozen Extractor' denotes fixed parameters. Columns 'A' and 'V' represent audio and video inputs, respectively, while a value of '0' for the other modality input indicates its absence. For trainable feature extractors, we introduce additional columns 'A(uni)' and 'V(uni)' to evaluate the direct use of extracted features for classification, thereby assessing feature extraction efficacy.

UAVM [6] emphasizes unified expression, highlighting the importance of modality absence. In contrast, AVoiD-DF [15] and MBT [3] prioritize exchanging feature semantics, making them particularly sensitive to missing modalities; MBT further distinguishes itself through the incorporation of bottlenecks. Notably, DrFuse [8] and MISA [16] marginally outperform our method, possibly due to the abundance of learnable cross-modal parameters enabled by their self-attention modules, which also magnifies the impact of modality absence. Perceiver [4], characterized by stacked features without explicit modal differentiation, is notably susceptible to missing modalities. In cases where the feature extractor is trainable, the impact of modality absence becomes more pronounced. At this juncture, this influence arises not only from modal fusion but also from the homogenization of features extracted by the feature extractor. GMF stands out for its minimal parameters and computational load, yet it achieves competitive performance while significantly reducing sensitivity to modality absence. This remarkable trait can be harnessed by integrating it with other methods, imparting them with similar characteristics. This integration leads to performance enhancement and decreased sensitivity to modality absence, showcasing the versatility and applicability of GMF.

For NMTs, our performance report on the ActivityNet [24] dataset is presented in Table 3. To be fair, we utilize the features same as AP-IVR [14] (4096-dimensional for video, 128-dimensional for images) as input. We map image features to 4096 dimensions or video features to 128 dimensions. Three combinations are obtained: Image-Video feature dimensions are (1) 128-4096 (denoted as 128-4096), (2) 4096-4096 (denoted as 4096), and (3) 128-128 (denoted as 128).

We employ CLIP [22] as the baseline, which only requires computing cosine similarity of mapped features without introducing parameters. METER [13] introduces the cross-attention module on this

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

basis, but the improvement is limited due to the sparse features. MAP-IVR [5] employs fixed-length mappings, while Perceiver [4] inputs an indistinguishable feature mapping, so the actual number of parameters relative to input dimensions is not apparent. GMF achieving competitive performance in (128) with minimal additional parameters and computations. Furthermore, the experiments (1284096) demonstrate the necessity of unequal-length fusion, ensuring not only the flexibility of the method but also profoundly impacting its performance and additional parameters. In the experiments of unequal-length fusion, GMF achieved state-of-the-art performance. Given that GMF is composed of linear layers, an increase in input dimensionality leads to an escalation in parameter count.

We performed a theoretical performance evaluation on FakeAVCeleb [26], as shown in Table 4. We use a feature extractor that is more compatible with the proposed method and remove the linear layer, denote as GMF-MAE (in Appendix, Fig. 14). For other SOTA methods involved in the comparison, we choose the feature extractor proposed in the original paper as much as possible (MISA utilizes sLSTM [30], UAVM adopts ConvNeXT-B [31], GMF-MAE employs MAE [32, 33]). The remaining methods, including Baseline employs R(2+1)D-18 [28]. Due to the imbalance in the dataset, with a ratio of approximately 1:39, the audio ratio is 1:1 and the video ratio is 1:19. UA VM [6] learns a unified representation, thus the easier classification of audio significantly impacts the overall results. Both DrFuse [8] and MISA [16] perform below our expectations; one potential explanation could be the influence of sample imbalance on their performance.

The performance of GMF remains consistent with the conclusions drawn from Table 2. Furthermore, GMF's insensitivity to missing modalities effectively mitigates the impact of sample imbalance, avoiding an excessive emphasis on any particular modality. The combination of GMF and MAE [32, 33] demonstrates optimal performance limits, validating our approach's effectiveness in addressing the challenges posed by downstream tasks. We provide a more comprehensive comparison with methods focused on deepfake detection in Table 7 (in Appendix).

6 Conclusion

In this paper, we combine the PNP equation with information entropy theory to introduce a multimodal fusion method for unrelated input features and downstream task features. The aim is to reduce the joint entropy of input features while decreasing the downstream task-related information entropy. Experimental results demonstrate that the proposed method takes a step forward in the generalization and robustness of multimodal tasks. Meanwhile, the additional burden can be negligible.

GMF comprises basic linear layers and is consequently susceptible to the inherent characteristics of linear operations, which exhibit growth in parameter count relative to input dimensionality. However, as per our theoretical framework, the effective component is proportional to the feature dimension. In forthcoming research, we intend to concentrate on sparsifying mapping matrices to further diminish parameter count.

329

330

331

332

333

334

335

336

337

338

339

340

341

342

References

Liang, V. W., Y. Zhang, Y. Kwon, et al. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Advances in Neural Information Processing Systems , pages 17612-17625. 2022.

Xia, Y., H. Huang, J. Zhu, et al. Achieving cross modal generalization with multimodal unified representation. In Conference and Workshop on Neural Information Processing Systems , pages 63529-63541. 2023.

Nagrani, A., S. Yang, A. Arnab, et al. Attention bottlenecks for multimodal fusion. In Conference and Workshop on Neural Information Processing Systems , pages 14200-14213. 2021.

Jaegle, A., F. Gimeno, A. Brock, et al. Perceiver: General perception with iterative attention. In International Conference on Machine Learning . 2021.

Liu, L., J. Li, L. Niu, et al. Activity image-to-video retrieval by disentangling appearance and motion. In Association for the Advancement of Artificial Intelligence . 2021.

Gong, Y., A. H. Liu, A. Rouditchenko, et al. Uavm: Towards unifying audio and visual models. 343 IEEE Signal Processing Letters , pages 2437-2441, 2022. 344

Ying, X. An overview of overfitting and its solutions. Journal of Physics: Conference Series , 345 page 022022, 2019. 346

347

348

349

Yao, W., K. Yin, W. K. Cheung, et al. Drfuse: Learning disentangled representation for clinical multi-modal fusion with missing modality and modal inconsistency. In Association for the Advancement of Artificial Intelligence , pages 16416-16424. 2024.

350

351

352

353

354

355

356

Ma, M., J. Ren, L. Zhao, et al. Are multimodal transformers robust to missing modality? In IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022.

Wang, H., Y. Chen, C. Ma, et al. Multi-modal learning with missing modality via shared-specific feature modelling. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15878-15887. 2023.

Kim, W., B. Son, I. Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning . 2021.

Li, J., R. R. Selvaraju, A. D. Gotmare, et al. Align before fuse: Vision and language representa357 tion learning with momentum distillation. In NeurIPS , pages 9694-9705. 2021. 358

Dou, Z.-Y., Y. Xu, Z. Gan, et al. An empirical study of training end-to-end vision-and-language 359 transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 360 18145-18155. 2021. 361

Xu, R., L. Niu, J. Zhang, et al. A proposal-based approach for activity image-to-video retrieval. 362 In Association for the Advancement of Artificial Intelligence , pages 12524-12531. 2020. 363

Yang, W., X. Zhou, Z. Chen, et al. Avoid-df: Audio-visual joint learning for detecting deepfake. 364 IEEE Transactions on Information Forensics and Security , pages 2015-2029, 2023. 365

Hazarika, D., R. Zimmermann, S. Poria. Misa: Modality-invariant and -specific representations 366 for multimodal sentiment analysis. In ACM International Conference on Multimedia , pages 367 1122-1131. 2020. 368

Wang, F., H. Liu. Understanding the behaviour of contrastive loss. pages 2495-2504. 2020. 369

Boudiaf, M., J. Rony, I. M. Ziko, et al. A unifying mutual information view of metric learning: 370 Cross-entropy vs. pairwise losses. In European Conference on Computer Vision , page 548-564. 371 2020. 372

373

Shang, Z. W., W. Li, M. Gao, et al. An intelligent fault diagnosis method of multi-scale deep

374

375

376

377

feature fusion based on information entropy.

Chinese Journal of Mechanical Engineering

, 2021.

Jiang, Q., C. Chen, H. Zhao, et al. Understanding and constructing latent modality structures in multi-modal representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7661-7671. 2023.

Granada, J. R. G., V. A. Kovtunenko. Entropy method for generalized poisson-nernst-planck 378 equations. Analysis and Mathematical Physics , pages 603-619, 2018. 379

[19]

Radford, A., J. W. Kim, C. Hallacy, et al. Learning transferable visual models from natural 380 language supervision. In International Conference on Machine Learning , pages 8748-8763. 381 2021. 382

Frankle, J., M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 383 In International Conference on Learning Representations . 2019. 384

385

386

387

Heilbron, F. C., V. Escorcia, B. Ghanem, et al. Activitynet: A large-scale video benchmark for human activity understanding. In 2015 IEEE Conference on Computer Vision and Pattern Recognition , pages 961-970. 2015.

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

Chen, H., W. Xie, A. Vedaldi, et al. Vggsound: A large-scale audio-visual dataset. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 721-725. 2020.

Khalid, H., S. Tariq, M. Kim, et al. Fakeavceleb: A novel audio-video multimodal deepfake dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks . 2021.

Vaswani, A., N. M. Shazeer, N. Parmar, et al. Attention is all you need. In Neural Information Processing Systems . 2017.

Tran, D., H. Wang, L. Torresani, et al. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6450-6459. 2018.

He, K., X. Zhang, S. Ren, et al. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778. 2016.

Hochreiter, S., J. Schmidhuber. Long short-term memory. Neural Computation , pages 17351780, 1997.

Todi, A., N. Narula, M. Sharma, et al. Convnext: A contemporary architecture for convolutional neural networks for image classification. In 2023 3rd International Conference on Innovative Sustainable Computational Technologies (CISCT) , pages 1-6. 2023.

He, K., X. Chen, S. Xie, et al. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009. 2022.

Huang, P.-Y., H. Xu, J. B. Li, et al. Masked autoencoders that listen. In Advances in Neural Information Processing Systems . 2022.

Kingma, D. P., M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations . 2014.

Hinton, G. E., R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science , pages 504-507, 2006.

Mittal, T., U. Bhattacharya, R. Chandra, et al. Emotions don't lie: An audio-visual deepfake detection method using affective cues. In Proceedings of the 28th ACM International Conference on Multimedia , page 2823-2832. 2020.

Nagrani, A., S. Albanie, A. Zisserman. Seeing voices and hearing faces: Cross-modal biometric matching. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8427-8436. 2018.

Hu, Y., C. Chen, R. Li, et al. Mir-gan: Refining frame-level modality-invariant representations with adversarial network for audio-visual speech recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 11610-11625. 2023.

Shankar, S., L. Thompson, M. Fiterau. Progressive fusion for multimodal integration, 2024. 425

426

Huang, G., Z. Liu, L. Van Der Maaten, et al. Densely connected convolutional networks. In

427

428

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

2017.

Skoog, D., D. West, F. Holler, et al. Fundamentals of Analytical Chemistry . 2021. 429

Nakkiran, P., G. Kaplun, Y. Bansal, et al. Deep double descent: Where bigger models and more 430 data hurt. In International Conference on Learning Representations , page 124003. 2020. 431

, pages 2261-2269.

[40]

Krizhevsky, A. Learning multiple layers of features from tiny images, 2009. 432

Howard, A., M. Sandler, B. Chen, et al. Searching for mobilenetv3. In 2019 IEEE/CVF 433 International Conference on Computer Vision , pages 1314-1324. 2019. 434

Dosovitskiy, A., L. Beyer, A. Kolesnikov, et al. An image is worth 16x16 words: Transformers 435 for image recognition at scale. In International Conference on Learning Representations . 2021. 436

Han, S., J. Pool, J. Tran, et al. Learning both weights and connections for efficient neural 437 network. In Advances in Neural Information Processing Systems . 2015. 438

Paszke, A., S. Gross, F. Massa, et al. Pytorch: An imperative style, high-performance deep 439 learning library. In Advances in Neural Information Processing Systems , pages 8024-8035. 440 2019. 441

Deng, J., W. Dong, R. Socher, et al. Imagenet: A large-scale hierarchical image database. In 442 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255. 2009. 443

444

445

Zhou, Y., S.-N. Lim. Joint audio-visual deepfake detection. In IEEE International Conference on Computer Vision , pages 14780-14789. 2021.

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

Cheng, H., Y. Guo, T. Wang, et al. Voice-face homogeneity tells deepfake. ACM Transactions on Multimedia Computing, Communications, and Applications , 2023.

Mittal, T., U. Bhattacharya, R. Chandra, et al. Emotions don't lie: An audio-visual deepfake detection method using affective cues. In ACM International Conference on Multimedia , page 2823-2832. 2020.

Zadeh, A., P. P. Liang, N. Mazumder, et al. Memory fusion network for multi-view sequential learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence , page 5634-5641. 2018.

Chugh, K., P. Gupta, A. Dhall, et al. Not made for each other- audio-visual dissonance-based deepfake detection and localization. In ACM International Conference on Multimedia , page 439-447. 2020.

Hara, K., H. Kataoka, Y. Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6546-6555. 2018.

Appendix / supplemental material 461

462

A Gradient Backward Flow

463

A.1 Definition and Explaination

In the gradient backward stage, the gradient is propagated from the output to the input direction 464 according to the adjustment of the downstream task loss. The specific gradient backward diagram is

Extraction

Fusion

Classification

Proposed objective

Grad of fusion loss

Grad of task loss

shown in Figure 3. The gradient generated by the downstream task loss is propagated through the entire network to the input, and the gradient of the fusion stage loss (if any) is propagated from the fusion output feature to the downstream task. The parameter adjustment of the feature extractor is affected by the gradient backward of the loss in the fusion stage and the loss of the downstream task.

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

It is worth discussing that the gradient adjustments from downstream task classification loss and fusion-related loss may not necessarily align. Hence, there typically exists a set of hyperparameters to balance the impacts of different losses. For instance, in V AE [34], the KL divergence loss and the reconstruction loss serve distinct purposes. The KL divergence loss facilitates model generalization, a significant divergence between VAE and AE [35], while the reconstruction loss is task-specific, reconstructing a sample from the latent space. However, both the KL divergence loss and the reconstruction loss in V AE often cannot simultaneously be zero. The KL divergence loss encourages some randomness in the latent space features, whereas the reconstruction loss favors more consistency in the latent space features. This balancing act is commendable, yet weighting between the losses poses a significant challenge. Hence, when all losses in multi-stage learning bear significance and the gradient descent directions of feature extractors are incongruent, balancing a hyperparameter becomes necessary to harmonize diverse learning objectives.

However, not all losses bear significance. Take contrastive loss, for example. It is a downstream 482 task loss in some NMT tasks [22], yet in most EMT tasks, contrastive loss typically operates in 483 the fusion stage, complementing downstream task-relevant cross-entropy losses, to narrow the gap 484 between positive samples in the latent space and push away negative samples. Some studies [1, 20] 485 have demonstrated the existence of gaps between modalities, and smaller gaps are not necessarily 486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

better. There are also analyses of the behavior of contrastive loss [17], aiming to minimize mutual information for positive sample pairs and maximize mutual information for negative sample pairs [18].

In EMT tasks, if positive and negative sample pairs coexist, as in Audio-Visual Deepfake Detection [15], the contrastive loss in the fusion stage aims to extract consistent information from positive sample pairs (representing real samples) while ensuring inconsistency in negative sample pairs (representing fake samples). It must be emphasized that the significant advantage of EMT tasks lies in modality commonality. Some studies have proven the existence of commonality [36, 37], but this doesn't alter the fact that auditory and visual modalities are fundamentally distinct (not only the semantic gap), with their enriched information not entirely consistent. In action recognition tasks, there is currently no work that effectively achieves this through audio; in speech recognition tasks [38], even with more complex, advanced feature extractors for extracting video features, or introducing priors to isolate video features solely for lip movements, the results are far inferior to audio single modality. While contrastive loss constrains the feature extractor to extract the most effective synchronous-related features, in the absence of a modality [15], it leads to a significant performance decline.

Moreover, not all tasks in EMT tasks involve positive and negative sample contrastive learning, so sometimes contrastive loss is equivalent to operating mutual information. For example, in some EMT methods' decoupling works [8, 16], each modality enjoys a common encoder and a specific encoder, minimizing mutual information for different modalities' common encoders to homogenize the extracted content and maximizing mutual information for the same modality's common encoder and specific encoder to heterogenize them, adapting well to the environment of modality absence. However, this method fixes the dimensions of each feature part, and the introduced losses directly manipulate the behavior of the feature extractor, compelling it to extract a predetermined quantity of common and specific features. The design of hyperparameters (encoder dimensions) will alter the behavior of the feature extractor. Additionally, when expanding to more modalities, the training cost of this method is also worth discussing.

A.2 Combine With Residual

ResNet [29] solves the bottleneck of the number of network layers, and this epoch-making work 514 allows the number of network layers to be stacked into thousands. A plausible explanation is that 515 it reduces gradient disappearance or gradient explosion in deep networks. We try to explain this 516 problem based on our information entropy related theory (Theory 3.1).

DownSample

Residual × n

Residual Forward

Residual Backward

Directly Backward

Directly Forward

Forward Grad. Sum.

Totally Forward

The basic block structure of ResNet [29] and the gradient propagation are illustrated in Figure 4. We 518 abstract it into a more general structure, where the downsampling block is considered as an arbitrary 519 function f ( · , θ f ) , and the residual block is considered as another arbitrary function g ( · , θ g ) . Here, 520 both of these arbitrary functions represent a type of network structure (in fact, this structure can be 521 further generalized), with θ f and θ g representing the parameters of the functions f and g , respectively. 522 Same as Eq.(2), the objective of gradient optimization is to optimize these parameters to minimize 523 the conditional entropy of Input X,Y and Output Y pred : 524

The expression for gradient descent can be derived by computing the partial derivatives of the loss function with respect to the parameters θ f and θ g . Denote the loss function as Eq.( 19), the gradient descent expressions are:

For functions f positioned further back, their ultimate gradients are influenced by the partial derivatives of the loss function with respect to functions g positioned earlier. If network g is composed of g 1 , g 2 , ..., g n , then during backward, it will be multiplied by numerous coefficients, making it more prone to gradient vanishing or exploding. The introduction of residuals can alleviate this problem. It is expressed as:

These derivatives represent the directions of steepest descent with respect to the parameters θ f and θ g , guiding the optimization process towards minimizing the loss function. Rethinking the associated gradient of f :

For the two elements of addition, compared to no residual, the first half of the gradient is numerically consistent, and the second half of the gradient is used as the residual. Obviously, this gradient is going to be direct.

Even in multimodal tasks, there exist challenges akin to residual issues yet to be resolved [39]. For instance, the association between feature extractors and downstream tasks may be compromised by the presence of feature fusion modules, manifested particularly in the introduction of intermediate gradients by deep fusion mechanisms, leading to gradient explosion or vanishing gradients. One approach to addressing this is through the incorporation of residuals. Indeed, some experimental endeavors have already undertaken this step, demonstrating its efficacy. These inferences may serve as a possible explanation, offering a generalized perspective.

However, residuals alone cannot entirely resolve the issue. Residuals, as a vector addition method, demand strict consistency in dimensions between inputs and outputs; moreover, excessive layer-bylayer transmission of residuals may result in the accumulation of low-level semantics onto high-level semantics, thereby blurring the representations learned by intermediate layers. While it may be feasible to employ residuals in a smaller phase within the fusion stage, utilizing residuals across the entire stage not only imposes stringent constraints on inputs and outputs but also risks semantic ambiguity.

Another method of applying residuals is akin to DenseNet [40], directly stacking channels. This still necessitates consistency in residual dimensions across different stages but circumvents the issue of semantic confusion. However, the final classifier remains a linear layer, requiring the flattening of multiple channels. Based on our theory, regardless of semantic sophistication, their initial origins remain consistent. As dimensions accumulate, elements describing the same set of features proliferate, inevitably leading to mutual information and subsequently reducing the conditional entropy relevant to downstream tasks.

In light of the foregoing analysis, residual connections at the skip-fusion stage can effectively alleviate the prevalent gradient issues in deep networks. However, this phased residual connection directly linking feature extractors to downstream tasks rigorously constrains the form of inputs and outputs, necessitating equilength features and overly blurred semantics, thus failing to achieve optimal effects. Furthermore, the nature of multimodal tasks diverges from simple downsampling-residual networks,

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

as gradients stem not only from downstream tasks but also from multiple sources before the fusion 565 stage. Our proposed method entails reducing the network layers in the fusion stage to align the fusion 566 gradients with the descent direction of downstream task gradients. Alternatively, the scope of the 567 fusion stage loss function gradient can be restricted. 568

B Proof of Theorem 3.4 569

We explain the derivation of the PNP equation to the proposed loss in detail. As before, let's assume 570 that the cell is one-dimensional, and only the direction x exists.

-

+

-

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

Voltage (also stimulation)

571

For the basic Nernst-Planck equation, as shown in Figure 5, the ion p ∈ { + , -} in the cell system 572 conforms to: 573

We abstract the feature vector into a one-dimensional electrolytic cell and need to correspond each 574 term of the equation to it. Throughout the system, the fluid remains stationary; The electric field E 575 that guides the movement of ions is generated by the electric potential ϕ and the magnetic field A . 576 We need to externally excite ϕ and do not additionally apply a magnetic field. The actual learning 577 rate is usually not very large (< 100), and the charge of the ion is assumed to be very small. This 578 gradient can be neglected as the magnetic field generated by the excitation. 579

Our external excitation electric field is constant, so the potential expression can be expressed by the 580 ion concentration. 581

The final state of the system is that the flux is fixed with respect to time, that is, the partial differential 582 is zero. From the ion point of view, diffusion and electromigration are in equilibrium. 583

Initial distribution

Ideal distribution

Actual distribution

In the initial condition, ions undergo spontaneous and uniform distribution through diffusion driven 584 by Brownian motion (the green line in Figure 6). This dynamic process leads to the establishment of a 585 heterogeneous distribution of ions within the system. However, as the system approaches the potential 586 equilibrium boundary b , the electrostatic forces acting on ions become increasingly influential. At 587 this boundary, denoted as the end condition [41], the principles of electroneutrality come into play. 588 Here, positive and negative ions are balanced such that their net charge is neutral, resulting in an 589 electrically neutral region around the potential equilibrium boundary: 590

591

592

593

594

595

The positive and negative properties of diffusion and electromigration are always opposite. If the ion species used as the external electrode is the same as that of the original solution, then we can approximately assume that the ion on either side of the zero potential boundary b , combined with the ion equivalent to the external electrode, can reduce the initial solute.

Assuming features from another modality are perfectly ordered, they can serve as a constant stimulus 596 guiding the ionization of the awaiting electrolytic modality. However, unlike in deep learning, where 597 the loss function can be equivalent to an external potential, both serve as stimuli capable of guiding 598 the respective fundamental ion directional motion. 599

Beginning with two modalities, initially disordered features prompt GMF to attempt cyclic con600 nections, as depicted in the diagram. The imposition of external guidance induces the movement 601 of feature particles of different polarities in distinct directions, ultimately coalescing at one end. 602

According to the law of conservation of mass, these aggregated features can be fully reconstructed 603 into the original modality representation of the guided modality particles at the opposite end. 604

Expanding to multiple modalities, electrochemical cells allow for parallel multi-level connectivity, 605 where applying a set of stimuli can simultaneously guide the movement of ions across multiple cell 606 groups. These potentials, as per the principles of basic circuitry, are distributed across each cell, as 607 shown in Figure 7.

-

+

-

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

+

-

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

+

-

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

608

The PNP equation provides a theoretical basis for GMF, and then we can propose to model material 609 conservation with a reconstruction loss. The reconstruction loss can well simulate the motion of 610 particles, and its reduction condition does not lead to ambiguity due to the existence of modality611 specific, as expressed in Eq.( 31). 612

613

2

4

8

16

32

64

128

70

75

80

85

90

95

ResNet20

ResNet32

ResNet44

ResNet56

ResNet110

(a) ResNet in CIFAR-10, Test ACC

2

4

8

16

32

64

128

0.92

0.94

0.96

0.98

1.00

1.02

1.04

ResNet20

ResNet32

ResNet44

ResNet56

ResNet110

(b) ResNet in CIFAR-10, Ratio

2

4

8

16

32

64

128

80

82

84

86

88

90

92

94

small

large

(c) MobileNet in CIFAR-10, Test ACC

2

4

8

16

32

64

128

0.90

0.92

0.94

0.96

0.98

1.00

small

large

(d) MobileNet in CIFAR-10, Ratio

32

64

128

256

512

1024

76

78

80

82

84

86

VIT7,Head=8

VIT12,Head=8

VIT12,Head=16

(e) ViT in CIFAR-10, Test ACC

32

64

128

256

512

1024

0.76

0.78

0.80

0.82

0.84

0.86

VIT7,Head=8

VIT12,Head=8

VIT12,Head=16

(f) ViT in CIFAR-10, Ratio

C Proof of Theorem 3.2

Theorem 3.2: The dimension of the feature that is best suited to the downstream task varies, and 614 there is always an optimal value for this feature. The dimension multiple relationship between each 615 layer of the feature extractor is fixed, and the initial dimension is adjusted. Too low dimension of the 616 final output will lead to inefficient representation, and too high dimension will introduce noise. The 617 existence of an integer l best such that for any integer l distinct from l best, the conditional entropy of 618 the model's predictions f l ( X,θ l ) is greater than that of the model's predictions f l best ( X,θ l best ) . 619

̸

620

C.1 Experiment

There was some previous work [42] that demonstrated that this optimal dimension exists. However, 621 existing methods do not account particularly well for the conditions under which poor fitting occurs, 622 so we conduct experiments to demonstrate the existence of this phenomenon. At the end we present 623 a possible conjecture. The existence of this optimal dimension is universal and at the same time 624 inconsistent. Specifically, each type of feature extractor, each type of dataset, and each corresponding 625 downstream task have different optimal dimensions. 626

Our set of experiments is shown in Fig 8. In addition to the intuitive visualization of the validation 627 accuracy, we also show the ratio of the validation accuracy to the training accuracy, aiming to measure 628 the validation accuracy and reflect the fitting effect of the model. The closer the ratio is to 1, the 629 stronger the generalization ability is, and the better the fit is. 630

631

632

633

634

635

636

637

In Figure 8 (a) and (b), the evaluation results of ResNet [29] on CIFAR-10 [43] are presented. As the dimensionality increases, the testing performance of the model improves, and the performance range stabilizes. However, with a twofold increase in dimensionality, the variation in testing performance diminishes, approaching zero. In other words, doubling the parameter count does not yield any improvement. Additionally, for larger networks like ResNet110, performance begins to decline. Furthermore, while absolute performance is increasing, the ratio is declining, indicating a weakening in generalization capability.

638

639

640

641

642

643

644

Figure 8 (c) and (d) depict the evaluation results of MobileNetV3 [44] on CIFAR-10 [43], showing conclusions similar to those of ResNet. For larger networks like MobileNetV3-Large, at lower dimensionalities, its generalization capability is significantly lower compared to simpler networks.

Figure 8 (e) and (f) illustrate the evaluation results of ViT [45] on CIFAR-10 [43]. As ViT is based on transformers [27] and possesses a global receptive field, its base dimensionality is significantly larger than that of convolutional neural networks. Both in terms of absolute performance and ratio, its optimal representation dimensionality approaches 256, distinct from other networks.

2

4

8

16

32

64

128

30

40

50

60

70

ResNet20

ResNet44

ResNet110

(a) ResNet in CIFAR-100, Test ACC

2

4

8

16

32

64

128

0.7

0.8

0.9

1.0

1.1

1.2

ResNet20

ResNet44

ResNet110

(b) ResNet in CIFAR-100, Ratio

However, it is worth noting that the presence of optimal features is not only closely related to network 645 type and structure, but also to the dataset and downstream tasks. We chose CIFAR-100 [43] for this 646 set of comparative experiments. This is because its data volume is consistent with CIFAR-10, but 647 with more categories and greater difficulty. The experimental results of ResNet [29] evaluated on 648 CIFAR-100 are shown in Figure 9. Compared to the results shown in Figure 8(a) and (b), firstly, the 649 impact of different dimensions on accuracy is more significant (for example, the maximum difference 650 in test performance of ResNet-20 on CIFAR-10 is about 25%, exceeding 40% here); for ResNet-110, 651 excessive dimensions no longer lead to performance stabilization, but rather a visible performance 652 decline. 653

654

655

656

The experimental results demonstrate the existence of an optimal dimensionality. This dimensionality may vary based on the different structures of networks. Hence, the concept of optimal dimensionality should be discussed in consideration of multiple external conditions.

657

658

659

660

661

662

663

D Proof of Theorem 3.3

Theorem 3.3: The feature extractor is fixed, and its original output feature dimension l is mapped to nl , and finally back to l . The mapping result is used as the basis for the downstream task. The performance of downstream tasks is infinitely close to the original performance as n increases, but never greater than the original performance. For magnification n > 1 , n ∈ Z , mapping matrix U 1 ∈ R l × nl and U 2 ∈ R nl × l , For the output features f ( X,θ ) ∈ R l and Y :

1

4

8

16

64

88

89

90

91

92

93

94

95

ResNet20

ResNet32

ResNet44

ResNet56

ResNet110

(a) ResNet in CIFAR-10, Test ACC

1

4

8

16

64

0.89

0.90

0.91

0.92

0.93

0.94

0.95

ResNet20

ResNet32

ResNet44

ResNet56

ResNet110

(b) ResNet in CIFAR-10, Ratio

D.1 Theoretically 664

Denote V = f ( X,θ ) , the rank of each stage: 665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

The mapped rank is always less than or equal to the original rank. That is, downstream task-relevant features may be compressed while not generating features out of thin air. Eq. (33) gets the certificate. For Eq.(34), we discuss the problem from pruning, linear algebra and probability theory. Neural networks are often overparameterized, requiring more network parameters than needed to get a good fit. In theory [23], however, only a subset of these parameters are useful in practice. Hence, some knowledge distillation methods such as teacher-student networks and pruning [46]. These tested models maintain good performance while removing most of the parameters, which proves that overparameterization is a common phenomenon. We interpret it as a probabilistic problem, that is, the effective parameters are generated with a certain probability. Overparameterization significantly improves the effective parameter generation, and knowledge distillation removes these redundant and invalid parameters.

Let A ∈ R nd × d be a learnable matrix ( n >> 1 ). Act on v ∈ R d to complete the mapping from lower dimension to higher dimension:

̸

Denote ˆ v ∈ R nd as the mapping result. ˆ v k represents the k -th row element. For any two of these row vectors a i and a j ( i = j ). They have a ratio c for their first elements.A necessary and sufficient condition for linearity between two vectors can be extended to the following: for any element in the same row of these two vectors, the ratio should be c .

If A is learnable, each element will have a different weight for each parameter adjustment. denote 685 P ( a jt a it = c ) as the probability that the proportion of the t -th element of a i and a j is equal to 686 c , which cannot be determined directly because the input sample is uncertain. In the context of 687 neural networks, the adjustment of gradients can be regarded as following a continuous probability 688 distribution. Consequently, the probability of the adjustment taking on a specific constant value 689

is zero (does not imply impossibility). By cumulatively multiplying this probability, we get the 690 probability that the two column vectors are linearly related in gradient descent. 691

However, for a d -dimensional vector, there cannot be more than d linearly independent features. To 692 simplify the expression, we assume that Eq.(40) is a fixed value on the interval (0,1). The probability 693 that exactly d-dimensional features are linearly dependent is given by: 694

695

In deep learning methods, the feature dimension is usually not set too small, d is sufficiently 696 large. Combined with gradient descent, the parameter adjustment is random, the linear correlation 697 probability of two random features is close to 0. 698

Consider mapping matrix U 2 ∈ R nl × l . As n increases, the probability of rank l increases. The same 699 is true for the matrix U 1 ∈ R l × nl . Therefore, as the probability of two correlation matrices being full 700 rank becomes larger, a larger n helps to restore the original representation under the premise that the 701 network does not involve unexpected situations such as gradient explosion and vanishing gradients. 702 However, it can be determined that when n is less than 1 (n > 0), there must be information loss. This 703 is because the upper limit of the rank of a matrix depends on the smaller value of the number of rows, 704 columns. Furthermore, it is not appropriate to increase the number of parameters blindly, which will 705 lead to an exponential number of parameters. 706

707

D.2 Experiment

64

128

256

512

1024

2048

4096

8192

16384

10

20

30

40

50

60

70

Val Acc

ResNet18

ResNet34

ResNet50

ResNet101

Dimention

We employed pre-trained ResNet-18, ResNet-34, ResNet-50, and ResNet-101 [29] models provided 708 by PyTorch [47], removing their classifiers to obtain raw features with dimensions of 512, 512, 2048, 709

and 2048 respectively. After freezing the other layers, we mapped these original features to another 710 dimension and subsequently retrained the classifiers based on these new features. As depicted in 711 Figure 11, where the abscissa represents the dimensions of the mapped features and the ordinate 712 represents the classification accuracy of the new classifier on the ImageNet [48] validation set. Our 713 experimental hyperparameter design and optimizer were identical to those reported in the original 714 paper. We recorded the validation accuracy every 400 iterations, and if the accuracy did not improve 715 for 10 consecutive validations, training was terminated prematurely. The final results are depicted in 716 a bar chart, where the upper and lower bounds represent the maximum and minimum values of the 717 validation accuracy. 718

719

720

721

722

723

It can be observed that larger mapping dimensions lead to faster convergence and yield better results. Smaller mapping dimensions, especially when they are smaller than the original dimensions, not only exhibit significant differences in upper and lower bounds of validation accuracy but also witness a substantial decrease in the upper limit. This observation aligns with our theoretical expectations. When the scaling factor n is close to 4, the performance loss has entered the acceptable range.

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

E Different Between Theorem 3.2 and Theorem 3.3

Both Theorem 3.2 and Theorem 3.3 focus on the dimension of presentation. The most significant difference between the two theories is what the original input was.

Theorem 3.2 is for the case where the sample is known and the representation is unknown, and this representation contains relevant information and irrelevant information. Therefore, this theory is more about the number of parameters needed to characterize, the minimum dimension needed to get the best performance, or the best performance in the minimum dimension. In this paper, this theory emphasizes the necessity of unequal-length fusion, and points out and proves through experiments that equal-length fusion may bring the problem of feature redundancy or feature missing, which not only increases the unnecessary amount of computation, but also affects the performance to some extent.

Theorem 3.3 is to analyze the influence of linear mapping on the representation in the case of known representation and unknown samples. Our proposed GMF method is very simple and contains only a number of linear layers, achieving the performance of larger parameter fusion methods of previous works. However, our original intention is not to be guided by experimental results, but to theoretically analyze whether the possible information loss is acceptable. We expect our work to be interpretable and applicable.

F Derivation of Conjecture 3.1

Conjecture 3.1: Rely on Theorem 3.1, 3.2, 3.3, we propose an conjecture that a boundary of performance limitation exists, determined by downstream-related entropy. Theoretically, by establishing a direct correspondence between the extractor and classifier, fusion method can enhance the limitation boundary, further improve performance.

Based on the proof of Theorem 3.2, one of the foundations of learning in neural networks is gradient descent, which presuppositions that gradients can be backpropagated. Every tuning of the learnable parameters will eventually be implemented on the original input. Assuming that the feature extractor is fixed, the original input at this time is the feature output by the feature extractor. For any learnable parameter, the value of a certain sample can be expressed by an exact formula. For a completely consistent input, it is assumed that its downstream task-related information entropy can be efficiently calculated, and its information entropy minimum is certain. Therefore, there is a performance upper bound, depending on how the existing features are utilized.

In practical deep learning tasks, the input features are often not fixed, and gradients need to propagate to be able to fully determine the original samples-which must also be fully determined. We continue to analyze the feature layers outputted by the feature extractor, assuming that the relevant information entropy of downstream tasks can be manually calculated. Thus, for the output features at a certain moment, the lower bound of the conditional entropy of downstream tasks can still be computed, which represents the performance upper bound.

Therefore, the entire multimodal learning network is divided into two parts: one is the lower bound of 760 the conditional entropy of the feature extractor output relative to the original samples, and the other 761 is the lower bound of the conditional entropy of downstream tasks relative to the feature extractor 762 output. The former is a prerequisite for the latter sequentially. However, as stated in the formulas, 763 assuming the existence of fusion loss and downstream task loss, and the gradient descent directions 764 are not completely consistent, let the weight of the fusion loss L fusion be λ 1 , and the loss of the 765 downstream task L task be λ 2 , the total loss can be expressed as: 766

The learning task is to minimize the training loss. Assuming that λ 1 L fusion > λ 2 L task , then the 767 gradient of the feature extractor will tend more toward the fusion loss. In severe cases (such as opposite 768 gradient descent directions), the downstream task-related loss will be completely overshadowed. 769 This also leads to an increase in the lower bound of the conditional entropy of downstream tasks 770 and a decrease in the theoretical performance upper limit. Therefore, we assume that there exists a 771 boundary, which is determined by the theoretical performance upper bound based on a fixed feature 772 and the conditional entropy of downstream tasks. Regardless of how outstanding the fusion method 773 design is, just like the principle of energy conservation law for features, the final task performance of 774 this method cannot exceed this upper bound. 775

Perceiver

GMF+Perceiver

GMF

Boundary of our proposed objective

Boundary of tranditional objective

Performance

Training step

The reason why our proposed GMF achieves performance improvement is not due to the performance 776 enhancement brought by the complex fusion network, but rather from a higher upper bound. However, 777 in reality, we are still far from this upper bound, and demonstrating our method as a precursor to 778 other methods can prove this point well. As shown in the Figure 12, we have drawn a hypothetical 779 graph based on the data reported in the paper. Assuming GMF as the precondition method for the 780 Perceiver [4], the result that GMF can be on par with complex networks with almost no resource 781 consumption is interpretable. 782

783

784

G Experiment Supplement

G.1 Implement Details

For all experiments, we use apex to optimize the v-memory and the parameter is set to 'O1'. The 785 random seed fixed '1' for all GMF related implementation. However, for some dropout design 786 methods, the reported experimental results may not be fully reproducible. More details are listed in 787 Table 5 788

(4) random.seed(seed) 792

G.2 Information About Preprocess and Baseline

For the VGGSound dataset, we downsample all currently available samples to 5fps, with videos of size 192*256 and audio sampled at 16000 Hz, while retaining only the first 9 seconds to accommodate most samples that are not exactly 10 seconds in duration. Samples without audio or video are removed. As for FakeAVCeleb, since the fabricated samples exhibit a global range of fabrication, with lengths distributed from 0.8 seconds and above, and a frame rate between 15 to 30 fps, we only select the first 8 frames along with their corresponding audio to ensure adaptability to the dataset.

We employ the default testing-training split provided by VGGSound. For FakeAVCeleb, consistent with much of the prior work focused on audio-visual deepfake detection, we first sort each class (real audio-real video, real audio-fake video, fake audio-real video, fake audio-fake video), and then allocate the first 70% of each class to the training set and the remaining 30% to the testing set.

The baseline of VGGSound pretrained on KINETICS400V1. Momentun of SGD = 0.9, weight delay=1e-4. Adam betas=(0.5, 0.9). lr_scheduler = ReduceLROnPlateau, factor=0.1, patience=1000 on VGGSound, factor=0.5, patience=50, verbose=True, min_lr=1e-8 on FakeAVCeleb. The generated audio sequence is quite long, and the receptive field of the convolutional network is not global. To address this potential issue, we stack the audio into a timing sequence (144000 to 9 × 16000).

Audio wave transform to input tensor by MelSpectrogram(sample_rate=16000, n_fft=400, win_length=400, hop_length=160, n_mels=192) for VGGSound and log (abs (STFT(n_fft=1024, hop_length=256, win_length=1024,window=blackman_window(1024))) + 1e-8) for FakeAVCeleb. Video frame directly as the input of network without any preprocess.

The hyperparameter as shown in Table 6

G.3 Compared Method Structure

The integration of our method with others is depicted in Figure 1. By bypassing modality-invariant features and focusing solely on modality-specific features for fusion, the input represents a representation with reduced mutual information. This leads to a reduction in the conditional entropy magnitude during the initial stages. The backend component may consist of a simple concatenation or modules proposed by other methods. Consequently, the inherent characteristics of GMF are constrained by the limitations of the backend module. Comparatively, the limitations are minimal with a simple concatenation approach.

H More Comparison on the FakeAVCeleb Dataset

Weexpanded the experimental table of FakeAVCeleb (Tab. 4) in the main text, incorporating additional comparisons focused on deepfake detection methods. Apart from the experiments reported in the

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

front-end

GMF

Extractor

Input

GMF

Output

back-end

Module

Task

Related

Module

Output

Final

Output

original text, the remaining data were sourced from the original paper proposing the method. Here, VFD [50], Emo-Foren [51], and MDS [53] are grouped together because these methods transform EMT into NMT. Specifically, these methods emphasize certain aspects of multimodal performance: VFD emphasizes identity, Emo-Foren emphasizes emotion, and MDS, while not emphasizing a specific mode, relies on computing confidence in matching a certain segment. Therefore, the modal absence evaluation for these methods is marked as '-', indicating absence. Importantly, our method effectively connects representations of different modalities without additional overhead for AE-based feature extractors, resulting in a highly competitive outcome.

H.1 The reason of choose FakeAVCeleb

The FakeAVCeleb dataset is atypical, characterized by severe class imbalance posing significant challenges to methods. Specifically, the ratio of positive to negative samples is 1:1 for audio and 1:19 for video, resulting in an overall ratio of 1:39. While audio often possesses discriminative capabilities less susceptible to the impact of sample proportions, most methods evaluated in our tests struggle to effectively address this bias.

Addressing this imbalance necessitates multimodal methods to learn weight disparities across modalities to mitigate the effects of sample bias. This manifests in high accuracy (ACC) juxtaposed with mismatched area under the curve (AUC). Methods capable of mitigating this bias often underutilize it, resulting in suboptimal ACC. However, in real-world scenarios, the distribution of genuine and fake

samples may not be balanced, and a single segment may not adequately represent an event. Hence, 843 the adaptability of methods to publicly available datasets warrants thorough investigation. 844

845

846

847

848

849

850

I GMFwith AutoEncoder (GMF-AE/MAE)

AutoEncoder [35] (AE) was initially proposed as a feature dimensionality reduction method, compressing samples into a latent space and then reconstructing them to retain the details of the entire sample in the latent space features. Masked AutoEncoder [32] (MAE) is a more powerful feature extraction variant of AE, masking most of the original samples and reconstructing them, allowing the model to learn more sample features. An intriguing point is that this concept can be seamlessly integrated with GMF (proposed Generalized Multimodal Fusion). GMF applies reconstruction loss

Because [Mask] are so [Mask]

Text

Audio

Visual

Because you are so beautiful

Output of Proposed GMF

invariant

specific

encoder

decoder

latent

X

X'

Mask

Reconstruct

Reconstruct

Reconstruct

Reconstruct

851 as an incentive, directing the movement of different types of features towards a relatively ordered 852 representation. Combining with PNP equations and our theoretical framework, this requires two 853 additional linear layers for feature dimensionality reduction, expansion, and a linear layer for recon854 struction. Thus, the additional overhead includes a reconstruction loss and the mentioned linear layers. 855 However, due to the nature of AE, this feature-directional movement process can be accomplished 856 during AE's self-supervised learning. Specifically, instead of feeding complete latent space features 857 into the Decoder, a combination of features from the corresponding Encoder and another modality 858 Encoder is used. This allows us to achieve our goal without any additional overhead. However, if 859 done so, explicit boundary delineation is necessary, which may affect model performance; moreover, 860

this learning process must be conducted in a multi-modal task, and features must be intact during the 861 learning process. 862

863

864

865

The specific structural diagram is shown in Figure 14. Here, we also consider the transformer [27] initially used for text as a variant of MAE, video encoder is MAE [32] and the Audio encoder is Audio-MAE [33].

866

J GMFArchitecture

Algorithm 1 GMF (Generalized Multimodal Fusion) 1: Input: Dimensions dims , multiple m , boundary b 2: Output: x 1 , x 2 , x 1 recon , x 2 recon 3: procedure GMF( x 1 , x 2 ) 4: x 1 inv , x 1 spec ← ELEMENTSPLIT( x 1 , dims [0] , min ( dims ) , m , b ) 5: x 2 inv , x 2 spec ← ELEMENTSPLIT( x 2 , dims [1] , min ( dims ) , m , b ) 6: x 1 ← concat ([ x 2 inv , x 1 spec ]) 7: x 2 ← concat ([ x 1 inv , x 2 spec ]) 8: x 1 re ← Linear ( x 1 , dims [0] + min ( dims ) , dims [0]) 9: x 2 re ← Linear ( x 2 , dims [1] + min ( dims ) , dims [1]) 10: return x 1 , x 2 , x 1 re , x 2 re 11: end procedure Algorithm 2 ElementSplit 1: Input: Dimension dim , min_len, multiple m , boundary b 2: Output: x inv , x spec 3: procedure ELEMENTSPLIT( x ) 4: b ←⌊ b × m × dim ⌋ 5: d ← m × dim 6: x ← Linear ( x, dim, m × dim ) 7: x inv ← Linear ( x [: , : b ] , b, min_len ) 8: x spec ← Linear ( x [: , b : d ] , d -b, dim ) 9: return x inv , x spec 10: end procedure Algorithm 3 Reconstruction Loss

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We propose a generalized multimodal fusion model via Poisson-Nernst-Planck Equation, which can greatly improve the fusion performance.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide an outlook on future work in the conclusion section and write a separate subsection in the appendix to state limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We briefly introduce the theory in the main text and present the necessary formulas that will help the reader to understand. For each proposed theory and hypothesis, the necessary derivations and experimental results are proved in the corresponding subsections of the experimental section and appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We fixed the random seed, and used source code that can be directly used as a class, rather than pseudocode, during the introduction to the algorithm. For fixed features, we will provide pre-trained models with the results of feature extraction. In addition, we have added our code in the attachment, and annotate the reference projects in detail. In addition, we present the detailed hyperparameters of the replication method in a tabular form in the appendix, and open source this part of the code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will open source all original code (such as the implementation and reproduction method of the proposed method), for non-original code, we will mark the reference project.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We briefly describe the hyperparameter Settings and experimental equipment used in the experiment section of the main text. For methods not previously available on the corresponding dataset, our preset hyperparameters are described in detail in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

1045

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

1059

1060

1061

1062

1063

1064

1065

1066

1067

1068

1069

1070

1071

1072

1073

1074

1075

1076

Justification: We fixed all random seeds to 1, and no data augmentation was applied to the original data, so the results should be similar across multiple runs. Furthermore, feature extractors are mostly aligned, which has nothing to do with dataset integrity.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: For each set of experiments in the main text, we report not only the evaluation metrics, but also the number of parameters, the amount of computation, and the time required for inference.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We've read the spec and followed it to the letter. Our paper does not involve human subjects and the datasets used are all open source datasets. These data sets are all instant downloads, and we cannot obtain them when the sample provider sets the sample private.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

1077

1078

1079

1080

1081

1082

1083

1084

1085

1086

1087

1088

1089

1090

1091

1092

1093

1094

1095

1096

1097

1098

1099

1100

1101

1102

1103

1104

1105

1106

1107

1108

1109

1110

1111

1112

1113

1114

1115

1116

1117

1118

1119

1120

1121

1122

1123

1124

1125

1126

1127

1128

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our approach is a deep learning architecture, and the selection of downstream tasks does not require a natural person to do it. This is not directly related to society.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The ultimate goal of our proposed method is not to propose any model, but to propose a valuable theory of multi-modal learning. The training data used only includes matching, classification and detection, and the data sets are all open source data sets. Therefore, as far as this article is concerned, there is no risk of abuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in 1129 the paper, properly credited and are the license and terms of use explicitly mentioned and 1130 properly respected? 1131

1132

1133

1134

1135

1136

1137

1138

1139

1140

1141

1142

1143

1144

1145

1146

1147

1148

1149

1150

1151

1152

1153

1154

1155

1156

1157

1158

1159

1160

1161

1162

1163

1164

1165

1166

1167

1168

1169

1170

1171

1172

1173

1174

1175

1176

1177

1178

1179

1180

Answer: [Yes]

Justification: We annotated any sources in detail.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: For the code and resources involved in the full text, we only provide our original parts, such as our methods and our reproduced methods. For resources that already exist (e.g., the feature extractor code), the reader should follow the documentation. Since the concept we propose contains some conclusions that should be tried by the reader (such as the optimal dimension), the specific training procedure should also be designed by the reader.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve any human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

1181

1182

1183

1184

1185

1186

1187

1188

1189

1190

1191

1192

1193

1194

1195

1196

1197

1198

1199

1200

1201

1202

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve any human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces CrossCheckGPT, a novel method for assessing hallucination robustness in multimodal foundation models without requiring reference standards. Utilizing cross-system consistency, the proposed method aims to provide a universal evaluation framework capable of being applied across various domains and tasks. This approach contrasts significantly with traditional hallucination assessments, which rely on comparison with gold-standard references and are limited to specific domains.\n\nSTRENGTHS:\n1. The introduction of a reference-free universal hallucination ranking method addresses a significant gap in the evaluation of foundation models, particularly in new and emerging areas.\n2. The paper effectively demonstrates the method's versatility across different modalities (text, image, and audio-visual), enhancing its relevance to a wide range of applications.\n3. The development of the AVHalluBench, the first audio-visual hallucination benchmark, is a noteworthy contribution that sets a new standard for evaluating models in this complex domain.\n\nWEAKNESSES:\n1. The analysis on how different models' outputs are compared and the implications of these comparisons could be more detailed. Specifically, the paper lacks a deeper exploration into the sensitivity of CrossCheckGPT to variations in model architecture or training data.\n2. Lack of additional visual representations of the data flow or examples of the hallucination checks.\n3. Lack of a more comprehensive set of benchmarks, including more direct comparisons with state-of-the-art methods.\n\nQUESTIONS:\n1. How does the CrossCheckGPT handle discrepancies in the quality of evidence models, especially when these models have different training backgrounds or data biases?\n2. Is there a quantitative measure of the 'distance' or difference between the outputs of various models that CrossCheckGPT uses to assess consistency?\n\nLIMITATIONS:\nThe effectiveness of the method hinges on the diversity and independence of the models used as evidence sources.  This dependence could pose challenges in scenarios where similar or homogenous models are prevalent.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

comments: SUMMARY:
The paper proposes a novel multimodal fusion method named Generalized Multimodal Fusion (GMF), which leverages the Poisson-Nernst-Planck (PNP) equation from physics to manage the feature fusion process in multimodal learning tasks. By treating features as charged particles, the method allows for a dynamic separation and recombination of modality-specific and modality-invariant features, thereby enhancing the fusion process and reducing the entropy in downstream tasks. This approach addresses common challenges in multimodal learning, such as feature dimension consistency, data integrity, and adaptability across various tasks.

STRENGTHS:
1. The application of the PNP equation, traditionally used in physics to describe the dynamics of charged particles, to multimodal feature fusion is highly original.
2. The paper is grounded in a solid theoretical framework that is well-articulated and robust.

WEAKNESSES:
1. The method, while innovative, appears to be complex in terms of implementation, particularly in how features are treated as charged particles. This complexity might limit its accessibility or usability for practitioners not familiar with the underlying physical equations.
2. The paper could benefit from more rigorous quantitative analysis, including statistical significance tests and error analysis. Such analyses would provide a clearer picture of the method's performance relative to benchmarks.
3. Can the authors provide the results on multimodal datasets with text-image modalities [1] ?

[1] Provable Dynamic Fusion for Low-Quality Multimodal Data.

QUESTIONS:
Please refer to the weakness.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a Generalized Multimodal Fusion (GMF) method using the Poisson-Nernst-Planck (PNP) equation to address challenges in multimodal fusion, such as feature extraction efficacy, data integrity, feature dimension consistency, and adaptability across various downstream tasks. The GMF method leverages theoretical insights from information entropy and gradient flow to optimize multimodal tasks, treating features as charged particles and managing their movement through dissociation, concentration, and reconstruction. \n\nKey contributions of the paper include:\n1. A theoretical framework combining PNP and information entropy to analyze multimodal fusion.\n2. A novel GMF method that dissociates features into modality-specific and modality-invariant subspaces.\n3. Experimental results showing GMF achieves competitive performance with fewer parameters on multimodal tasks like image-video retrieval and audio-video classification.\n\nSTRENGTHS:\nOriginality: The application of the PNP equation from physics to multimodal feature fusion is novel and creative. The theoretical framework combining PNP and information entropy provides an original perspective on analyzing multimodal learning.\n\nQuality: The paper provides a solid theoretical foundation with detailed proofs and derivations. The experimental evaluation is comprehensive, covering multiple datasets and task types (NMT, EMT, GMT).\n\nClarity: The paper is well-structured and clearly written. The methodology is explained step-by-step with helpful visualizations.\n\nSignificance: The proposed GMF method shows promising results in terms of performance, parameter efficiency, and robustness to missing modalities. It has potential for broad applicability as a frontend for other fusion methods.\n\nWEAKNESSES:\n1.The theoretical analysis, while extensive, could benefit from more intuitive explanations to improve accessibility.\n2. The experimental section lacks ablation studies to isolate the impact of different components of the GMF method.\n3. While the method shows good results, the performance improvements over some baselines are relatively small in certain experiments (e.g. Table 2).\n4. The paper does not thoroughly discuss potential limitations of the approach or scenarios where it may not be suitable.\n\nQUESTIONS:\n1. How sensitive is the GMF method to the choice of hyperparameters like the dissociation boundary b(j)? Was any systematic hyperparameter tuning performed?\n2. The paper mentions GMF can serve as a frontend for other modules. Were any experiments conducted combining GMF with more advanced fusion methods beyond MBT and Perceiver?\n3. How does the performance of GMF scale with increasing numbers of modalities beyond the audio-visual experiments presented?\n4. Are there any theoretical limitations on the types of features or modalities that can be fused using the PNP-based approach?\n\nLIMITATIONS:\nThe authors briefly mention some limitations of linear operations for high-dimensional inputs in the conclusion. However, a more thorough discussion of potential limitations and failure cases would strengthen the paper. Additionally, while not highly relevant for this theoretical/methodological work, some discussion of potential negative societal impacts of improved multimodal fusion techniques (e.g. privacy concerns) could be included.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
In this paper, the authors combined the Poisson-Nernst-Planck (PNP) equation with information entropy theory and proposed a generalized multimodal fusion approach, which disassociates modality-specific and modality-invariant features, thereby reducing the join entropy of input features and meanwhile decreasing the downstream task-related information entropy. The experimental results demonstrate that the proposed approach can improve the generalization and robustness of multimodal tasks.

STRENGTHS:
- It is innovative to employ PNP for solving the multimodal fusion issue, by treating features as charged particles to disassociate them. 
- A generalized multimodal fusion approach was designed, which can overcome the strong assumptions made by existing methods. 
- Experiments showed the effectiveness of the proposed GMF approach in efficiency and flexibility.

WEAKNESSES:
- Significance test (e.g., Wilcoxon signed-rank test) would be helpful to better illustrate the significance of the proposed method compared against baselines. 
- Social impacts are not explicitly discussed in the paper/appendix.

QUESTIONS:
- Section 5.1, the authors mentioned “For NMTs, ActivityNet dataset evaluate (4)”. However, there is not (4) found in the above context. 
- It is not quite how GMF works as frond-end when combing with other multimodal fusion approaches. 
- According to Table 2, GMF works better when taken as back-end compared against front-end. Any possible explanations for these results?

LIMITATIONS:
Please refer to Weakness and Question, which are the aspects suggested to be further improved.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13