ERL-MR: Harnessing the Power of Euler Feature Representations for Balanced Multi-modal Learning

Authors: weixiang  han
chengjun  cai
yu  guo
jialiang  peng
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 09:04:55

Abstract: Multi-modal learning leverages data from diverse perceptual media to obtain enriched representations, thereby empowering machine learning models to complete more complex tasks. However, recent research results indicate that multi-modal learning still suffers from “modality imbalance”: Certain modalities' contributions are suppressed by dominant ones, consequently constraining the overall performance enhancement of multimodal learning. To tackle this issue, current approaches attempt to mitigate modality competition in various ways, but their effectiveness is still limited. To this end, we propose an Euler Representation Learning-based Modality Rebalance (ERL-MR) strategy, which reshapes the underlying competitive relationships between modalities into mutually reinforcing win-win situations while maintaining stable feature optimization directions. Specifically, ERL-MR employs Euler's formula to map original features to complex space, constructing cooperatively enhanced non-redundant features for each modality, which helps reverse the situation of modality competition. Moreover, to counteract the performance degradation resulting from optimization drift among modalities, we propose a Multi-Modal Constrained (MMC) loss based on cosine similarity of complex feature phase and cross-entropy loss of individual modalities, guiding the optimization direction of the fusion network. Extensive experiments conducted on four multi-modal multimedia datasets and two task-specific multi-modal multimedia datasets demonstrate the superiority of our ERL-MR strategy over state-of-the-art baselines, achieving modality rebalancing and further performance improvements.

Main paper: 
 introduction: 
 The advent of the multi-modal learning paradigm [35, 43, 50] represents a pivotal advancement in artificial intelligence (AI) technology, elevating AI models to a new level of capability [7, 39, 58] . This paradigm empowers AI models with rich perceptual abilities, enabling them to undertake increasingly complex multimedia tasks, including sentiment analysis [14, 79] , audio-visual speech recognition [32, 78] , and visual question answering [29] , among others. Specifically, multi-modal learning endeavors to mine and analyze data from multiple sensory media, harnessing insights from diverse sources. This approach is widely recognized for its potential to enhance AI model performance in comparison to training with uni-modal data.However, recent research results [9, 16, 40, 60, 63, 68, 78] have revealed a counterintuitive fact: the learning processes within each modality of multi-modal learning exhibit a dynamic competition rather than the anticipated mutual benefit. As an illustration, Huang et al. in [16] devised a theoretical analytical framework to substantiate the aforementioned claims, subsequently yielding theoretical insights. Their research underscores that in some cases, the optimal uni-modal network outperforms the multi-modal network trained jointly, a finding which challenges conventional wisdom [35, 43, 50] . In this context, researchers attribute this phenomenon to the "modality imbalance" issues [9, 40, 68] , wherein a dominant modality restricts the comprehensive exploitation of the multiple modalities at play. This, in turn, undermines the fundamental objective of enhancing model performance through the fusion of information from diverse modalities.To further explain the reasons behind the above phenomena, a series of empirical and theoretical methods [9, 15, 16, 24, 40, 62, 63, 68] have been proposed to analyze the learning process of multi-modal learning. Some of them try to explain it from the perspective of joint training, that is, they believe that different modalities will compete with each other during the process of joint training the fusion network via gradient descent [15, 16] . More specifically, the competitive behavior materializes as the dominant modality converges rapidly with a substantial gradient amplitude, effectively impeding the convergence and optimization of the weaker modality [1, 10] . Another predominant perspective underscores the challenge posed by the heterogeneity of multi-modal data, specifically concerning the acquisition of knowledge related to multi-modal correlations and complementarities [9, 40, 63] . These modal correlations and complementarities are manifested through feature representations. However, heterogeneous feature representations can disrupt intermodal learning and result in incongruent learning efficiencies across different modalities.In light of the aforementioned insights, researchers are dedicated to the development of multi-modal learning strategies aimed at addressing the challenge of "modality imbalance. " To tackle the issue, prior works employ gradient-based methods, including gradient angle constraints [68] , on-the-fly gradient modulation [40] , and gradient balance constraints [63] . While these approaches effectively regulate the learning rates of various modalities through gradient adjustments and constraints, they may still fall short in preventing the dominant modality from interfering with the gradient update direction of the weaker modality [9, 15, 16] . This limitation poses a challenge in enhancing the performance of the slow-learning modality. Another set of methods concentrates on formulating efficient feature utilization techniques designed to mitigate the negative effects of heterogeneous multi-modal data on learning speed [3, 9, 22, 30, 38, 68, 70] . The key idea of their approach revolves around energizing the slower learning modality through facilitating more robust feature utilization, while concurrently mitigating the influence of the dominant modality by slowing down itself during the early stages of training. Clearly, the solely focusing on enhancing feature utilization and learning efficiency of the weaker modality may not yield benefits for the fusion network, as they impede the learning process of the dominant modality. Current Dilemma. The prevailing dilemma manifests itself as a delicate trade-off. When attempts are made to mitigate modality imbalance by restraining the optimization of dominant modalities, the result can be a decline in the performance of these strong modalities, consequently exerting a negative impact on the overall performance of the fusion model [9, 16] . On the other hand, if efforts are oriented toward stimulating slower-learning modalities to foster modality balance, it might entail a deceleration in the optimization pace of the dominant modalities, consequently diminishing the advantages they bring to the fusion network [15] . Hence, the key challenge lies in finding a solution to break through this dilemma and devise a way that can simultaneously tackle both issues effectively. Our Contributions. To address the aforementioned challenges, we explore a new path, one that eschews the traditional strategies of either suppressing dominant modalities or invigorating weaker ones to achieve modality balance and boost fusion network performance. Specifically, we propose an Euler Representation Learning-based Modality Rebalance (ERL-MR) strategy. In this strategy, our insight centers on the concept that the Euler vector space transformation of modal features can aptly capture the interplay between modalities, effectively reshaping their relationship from competition to mutual reinforcement. Furthermore, our observations indicate that imposing constraints on the feature phases between different modalities and optimizing the cross-entropy loss for each modality proves advantageous in enhancing both modality-specific performance and the overall performance of fusion network. Therefore, we have made the following designs: (D1.) -Euler Transformation Design. In this design, ERL-MR endeavors to transform the vector feature representations of diverse modalities into a complex vector space using Euler's formula, with the primary objective of enhancing modal features. Subsequently, we harness the geometric attributes of the Euler transformation to effectively reconfigure the relationship between modalities, transitioning it from competition to mutual reinforcement. The reason behind this approach is that within the complex space, our design has the capacity to model the relationships of phase and modulus associated with different modalities, thereby facilitating the integration of feature interactions among the modalities. (D2.) -Constrained Loss Design. In this design, ERL-MR aims to take into account the performance of each modality and the fusion model. To achieve this, we propose a joint loss, the multi-modal constrained loss function, including two critical components: the cosine similarity constraint, aligning optimization directions between modalities, and the cross-entropy loss constraint, indicating the correct optimization direction for each modality. In particular, in the cosine similarity constraint, we constrain the cosine value of the phase of the complex features between each modality, which is equivalent to performing a geometric transformation on complex features.The main contributions are summarized below:• We propose an effective modality rebalance strategy, i.e., ERL-MR, against modal imbalance in multi-modal learning. This strategy is model-agnostic and can naturally be integrated into the framework of multi-modal learning. • We leverage insights from multi-modal imbalance analysis to propose two simple yet effective designs: Euler transformation and multi-modal loss. Additionally, we provide a complexity analysis of the overall strategy. • We conduct extensive case studies on six real-world multimodal datasets, demonstrating that our strategy significantly outperforms existing state-of-the-art methods. 
related work 2.1 multi-modal learning: 
 Multi-modal learning aims to extract complementary or independent knowledge from various modalities, enabling the representation of multi-modal data [2, 28, 71] . This endeavor empowers AI models with the capability to comprehend and process diverse modal information [65] . Within this domain, a valuable research direction is how to efficiently extract and fuse meaningful representations [25, 31, 72] . For example, researchers leverage graph representations of multi-modal multimedia data to employ knowledge graph embedding techniques [6, 54, 61] , facilitating the accomplishment of diverse complex tasks, e.g., biological knowledge graph completion [66, 67] , knowledge graph reasoning [75] , and graph recommendation systems [42, 76, 77] . Furthermore, multimodal learning technique can serve as a foundational building block within a model to harness the potential of multi-modal information, leading to enhanced model performance in specific domains, e.g., traffic trajectory prediction [47, 48, 69] , disease diagnosis [4] , action recognition [5, 11, 33] , audio-visual speech recognition [32] , and visual question answering [29] . In recent times, multi-modal techniques have gained significant prominence within the domain of large language models (LLM), resulting in the development of multi-modal LLMs tailored for specific applications. Notable examples include the HuaTuo medical large model [56] , Llama 2 [53] ,GPT-4 vision model [36, 46] , and others, each designed for specific fields. However, as mentioned above, how to harness the power of each modality data in multi-modal learning is still an open question.Existing Rebalance Approaches and Their Limitations. One of the straightforward reasons hindering the realization of the full potential of each modality data is the issue of modality imbalance [5] .To address the problem of modality imbalance, researchers have devised a variety of techniques that can be categorized into two main groups: gradient-based methods [40, 60, 63, 68] and feature-based methods [3, 9, 38, 78] . For gradient-based methods, their insight is to suppress the convergence speed of the dominant modality through gradient modulation or constraints to achieve modality balance. Nevertheless, these methods may entail a reduction in the dominant mode's contribution to the fusion network, which is not conducive to enhancing the overall performance of the fusion network [9] . On the other hand, feature-based methods, as employed in prior research, concentrate on extracting the features of weaker modes to expedite their convergence for achieving modal balance. However, such methods do not change the nature of competition between modalities and cannot significantly improve the overall performance of the fusion network [16] . Hence, existing methods tend to address only a limited aspect of the problem, as they do not alter the fundamental competitive relationship between modalities, which prevents them from mutually benefiting each other.Euler representation learning is a machine learning technique designed to develop feature representation methods capable of directly processing geometric data, such as 3D shapes and graphics [27, 34, 57, 59, 64, 74] . Researchers have delved into approaches involving Euler feature representations to capture the inherent geometric characteristics of the data, enabling neural networks (e.g., graph neural networks (GNN) [8, 13, 23] ) to effectively handle geometry-related tasks. For example, Jiang et al. in [18] applied Euler features to characterize the geometric correlations among users on the Taobao platform. They harnessed the potent learning capabilities of GNN to craft a recommendation system capable of accommodating millions of users. Hence, Euler representation is frequently employed in the realm of representation learning and the learning of geometric topological structures, primarily due to its advantageous properties for feature enhancement and geometric feature transformation [3, 17, 26, 38, 49] . To this end, we aim toward balanced multi-modal learning by leveraging the special power of Euler feature representation in this regard. 
preliminaries 3.1 modality imbalance: 
 Consider a multi-modal dataset D, which includes 𝑀 distinct modalities, labeled as 𝑚 0 , 𝑚 1 , . . . , 𝑚 𝑀 -1 . In the Multi-modal learning, encoders𝜓 𝑚 1 (𝜔 𝑚 0 , •),𝜓 𝑚 1 (𝜔 𝑚 1 , •), . . . ,𝜓 𝑚 1 (𝜔 𝑚 𝑀 -1 ,•) to extract features from modality 𝑚 0 , 𝑚 1 , . . . , 𝑚 𝑀 -1 , respectively, where 𝜔 is the parameters of the encoder. Let 𝑓 represent the fusion model that has been trained on the dataset D. In this context, the performance of the multi-modal fusion model and each modality can be denoted as 𝑓 (D) and 𝑓 (𝑚 0 ), 𝑓 (𝑚 1 ), . . . , 𝑓 (𝑚 𝑀 -1 ), respectively. When the performance of a specific modality significantly exceeds that of other modalities, this situation is termed modality imbalance, with the particular modality 𝑚 𝑖 (𝑖 ∈ {1, 2, • • • , 𝑀 -1}) identified as the dominant modality. For each modality 𝑚 𝑖 , during each iteration 𝑡, if 𝑚 𝑖 is the dominant modality, the performance measure 𝑓 (𝑚 𝑖 ) will converge more rapidly compared to the other modalities. In addition, we have provided theoretical proof for the definition of Modality Imbalance [9] . The complete proof can be found in the full version.We briefly introduce the formal expression of Euler's formula and its application in complex vector spaces [52] . Euler's formula is a foundational mathematical equation in complex analysis, establishing a fundamental connection between trigonometric functions and the complex exponential function. Specifically, for any real number 𝑥, Euler's formula states:EQUATIONwhere 𝑒 is the base of the natural logarithm and 𝑖 is the imaginary unit. Eq. ( 1) can also be formulated as:EQUATIONwhere 𝝀𝑒 𝑖𝜽 and 𝝀 cos 𝜽 + 𝑖 (𝝀 sin 𝜽 ) represent the complex vector in polar and rectangular forms, respectively. Here, 𝝀 and 𝜽 are the modulus and phase of the complex vector. For a complex vector 𝒓 + 𝑖𝒑, we define the real part as 𝒓 = 𝝀 cos 𝜽 and the imaginary part as 𝒑 = 𝝀 sin 𝜽 . The modulus 𝝀 and phase 𝜽 can be expressed as follows:EQUATIONwhere atan2(•, •) denotes the 2-argument arctangent function. The application of Euler's formula enables the conversion of complex vectors from the rectangular form to the polar form, facilitating the encoding of features in the polar space. 
methodology: 
 In contrast to prior efforts that sought to address this problem by constraining the optimization speed of dominant modalities or hastening the training of weaker modalities, our objective is to delve into the intricate relationships between different modality features with the aim of establishing a state of mutual reinforcement. To this end, we design two components in the ERL-MR strategy: Euler Feature Transformation and Multi-modal Constrained Loss.The overview of the proposed strategy is shown in Fig. 1 . Complex Space Mapping. Our key insight revolves around the likelihood that the phenomenon of modality imbalance may be attributed to the heterogeneity within the feature spaces of different modalities. In this context, the first problem we face is how to map the heterogeneous feature space to a homogeneous feature space to alleviate the imbalance phenomenon. Inspired by Euler's formula (i.e., Eq. ( 2)), we aim to use it to implement feature mapping. Specifically, we map the feature embedding vectors produced by the modal encoder 𝜓 from real vector space to complex vector space via Euler's formula. Consider a feature embedding 𝒆 𝑗 = 𝜓 𝑚 𝑖 𝑗 (𝜔 𝑚 𝑖 , 𝑥 𝑚 𝑖 𝑖 ) 𝑁 𝑗=1 produced by the encoder of modality 𝑚 𝑖 . Following Eq. ( 2), we can map 𝜓 𝑚 𝑖 𝑗 (𝜔 𝑚 𝑖 , 𝑥 𝑚 𝑖 𝑖 ) into the complex vector space, representing it with two real vectors 𝒓 and 𝒑 to denote the real and imaginary parts of the complex vector, respectively. Thus, we have:EQUATIONwhere ẽ𝑗 is a feature embedding in the complex vector space. To transform feature embeddings into complex vectors, the fundamental concept is to regard them as phases and integrate learnable parameters 𝜇 𝑗 as moduli (i.e., 𝜇 𝑗 → 𝝀, 𝒆 𝑗 → 𝜽 ), following Euler's formula as presented in Eq. ( 2):EQUATIONFor simplicity, we let:EQUATIONwhere 𝑁 is the number of features. Thus, in order to simplify the expression, we can rewrite Eq. ( 5) as follows:EQUATIONInsight 1. In multi-modal feature representation learning, Euler's formula maps features to the complex vector space, preserving and sharing valuable information among different modalities. Moreover, the sine and cosine operations inherent in Euler's formula contribute to alleviating feature differences between different modalities by employing polar coordinate transformation. Therefore, our method aims at enhancing feature presentation and minimizing feature presentation disparities, effectively transforming the competitive relationship between modalities into a cooperative one. Feature Compression. Although complex space mappings offer feature-enhanced representations, their integration into the model architecture introduces challenges, notably increased computational costs. This aspect poses difficulties in the practical deployment of this strategy. To address this issue, we design a compression operation that uses a simple linear layer to compress the features obtained by the complex space mapping operation. Its formal definition is as follows:EQUATIONwhere ψ𝑚 𝑖 𝒓 (𝜔 𝑚 𝑖 , 𝑥 𝑚 𝑖 ) and ψ𝑚 𝑖 𝒑 (𝜔 𝑚 𝑖 , 𝑥 𝑚 𝑖 ) respectively represent the real part feature and imaginary part feature obtained after the compression operation,𝑊 𝑚 𝑖 𝒄𝒐𝒎𝒑𝒓 𝒆𝒔𝒔 ∈ R 𝑁 × 𝑑 𝝍 𝑚 𝑖 𝒓 (𝜔 𝑚 𝑖 ,𝑥 𝑚 𝑖 ) and 𝑏 𝑚 𝑖 𝒄𝒐𝒎𝒑𝒓 𝒆𝒔𝒔 ∈ R 𝑑 𝝍 𝑚 𝑖𝒓 (𝜔 𝑚 𝑖 ,𝑥 𝑚 𝑖 ) represent the parameters of the linear layer. After compressing the real and imaginary features, we apply a nonlinear activation function, i.e., ReLU, to generate the compressed feature representations via Eq. (9) .EQUATIONSubsequently, we execute a concatenation operation using Eq. (10) to merge the real and imaginary features into new features for model training. where ⊕ represents the concatenation operation, and 𝝍 𝑚 𝑖 𝑵 𝒆𝒘 (𝜔 𝑚 𝑖 , 𝑥 𝑚 𝑖 ) is the enhanced feature obtained by concatenating the real part feature and the imaginary part feature. It should be noted that the dimensions of 𝝍 𝑚 𝑖 𝑵 𝒆𝒘 (𝜔 𝑚 𝑖 , 𝑥 𝑚 𝑖 ) remain the same as the original features 𝝍 𝑚 𝑖 (𝜔 𝑚 𝑖 , 𝑥 𝑚 𝑖 ) to maintain the model structure and reduce computational costs.EQUATIONBy incorporating complex space feature mapping and enhancement, it refined feature representation facilitating collaborative training for each modality and establishing a mutually beneficial relationship. This cooperative approach mitigates the impact of modal imbalance to some extent. However, we continue to face the challenge of optimization direction drift, posing an obstacle to further enhancing the model's performance.To tackle the aforementioned challenge, we propose the Multimodal Constrained Loss (MMCLoss) to aid in adjusting the training direction of each modality. To implement this loss, we initially integrate the concept of phase, leveraging the distinctive attributes of the complex space. Based on Eq. ( 3) and Eq. ( 8), the phase 𝝋 𝑚 𝑖 of modality 𝑚 𝑖 is redefined as:EQUATIONInsight 2. Utilizing polar coordinate space allows for a deeper understanding of complex feature information and a clearer perception of feature vector directions compared to rectangular space.Here, we elaborate on how phase information can guide model optimization directions. Initially, we calculate the phase 𝝋 𝑚 𝑖 of each modality 𝑚 𝑖 using Eq. ( 11), which encapsulates the feature phase information of modality 𝑚 𝑖 . Moreover, as illustrated in Eq. ( 3), the eigenphase represents the direction indicated by the eigenvector in polar space. In the multi-modal training, we observe that the phase and optimization direction of each modality undergoes constant changes with increasing training epochs. Based on this observation, we contend that modal phase information can be leveraged to dynamically adjust the direction of different modality optimizations. Building on the prior discussion, in multi-modal learning, the dynamic phases act as indicators of the model's adaptive adjustments and optimization trajectory. To mitigate the performance degradation caused by drift in modal optimization directions, our objective is to align the optimization directions of each modality as closely as possible during training. To realize this objective, we introduce cosine similarity as a mechanism to constrain the feature phases:EQUATIONwhere we use 𝝋 𝑚 𝑖 and 𝝋 𝑚 𝑖 +1 to represent the phase of the two modalities for calculating L 𝑚 𝑖 Cosine-similarity . Through Eq. ( 12), we keep the optimization directions of modality 𝑚 𝑖 and mode 𝑚 𝑖+1 gradually closer during the training process of multi-modal learning. However, how to ensure the correctness of the constrained optimization direction and maintain the performance of each modality is another serious challenge.To this end, we propose a simple and efficient method in MM-CLosss. Specifically, we cleverly use cross-entropy loss to calculate the loss of logits of each modality:EQUATIONwhere L 𝑚 𝑖 𝐶𝐸 represents the cross-entropy loss of modality 𝑚 𝑖 . Through Eq. ( 12) (13), we form the MMCLoss as follows:EQUATION)Discussion. The MMCLoss includes two components: cosine similarity and cross-entropy. It is crucial to note that the cosine similarity part of MMCLoss differs from previous approaches [9, 68] that concentrated on constraining the angle of the gradient. In our design, we leverage the complex phase to constrain the optimization direction between different modalities. Additionally, MMCLoss incorporates the cross-entropy loss of each modality, ensuring that the training process of different modalities consistently follows the correct path, thereby fostering cooperation between modalities while avoiding biased optimization directions. In this manner, MM-CLoss facilitates the fusion model in achieving higher performance levels and maintaining modality balance from two perspectives. Furthermore, we offer additional elucidation from the standpoint of visualization and complex triangle inequality theory in the full version to support the conclusion that the ERL-MR strategy enhances correlation and complementarity among diverse modality data, effectively governing the optimization speed and direction of the model.We present a time complexity analysis of the proposed ERL-MR strategy. Furthermore, the complete algorithm for the ERL-MR strategy is shown in Algorithm 1 in the full version. For simplicity, we assume that a convolutional neural network is applied in the feature extraction process. In addition to the computation complexity of both convolution operation per sample and the gradient update, the complexity for Euler feature transformation and feature compression can be calculated as 𝑂 (1) + 𝑂 (2𝜁 2 ) where 𝜁 is the dimensionality of 𝜓 𝑚 𝑖 𝒓 (𝜔 𝑚 𝑖 , 𝑥 𝑚 𝑚 𝑖 ). For MMCLoss, the complexity of this operation is 𝑂 (𝑀) +𝑂 (𝑀 • 𝑁 •𝐶) due to the L Cosine-similarity and L 𝐶𝐸 involved. Therefore, the introduced additional complexity for the proposed ERL-MR strategy is 𝑂 (2𝜁 2 + 𝑀 + 𝑀 • 𝑁 • 𝐶). 
evaluation 5.1 experimental settings: 
 To evaluate the performance of our ERL-MR strategy, we conduct extensive experiments on six benchmarking datasets. All experiments are developed using Python 3.9 and PyTorch 1.12 and evaluated on a server with an NVIDIA A100 GPU. Datasets. We validated the ERL-MR strategy from multiple task perspectives using six multi-modal multimedia datasets in the experiment, including USC Dataset [73] , AVE Dataset [51] , MHAD Dataset, Colored-gray MNIST Dataset (CGM) [21] , FLASH Dataset [45] and Alzheimer's Disease Monitoring (ADM) Dataset [37] . Table 1 summarizes the attribute information of the various multi-modal benchmark datasets in the experiment, and the complete information is provided in the full version. Models. For the ADM dataset, we utilize the TDNN architecture [55] to extract audio features, while CNN layers are employed for extracting radar and depth image features. For USC, MHAD, and Flash datasets, we utilize a 2D-CNN model for the GPS data, whereas a 3D-CNN architecture is used to handle the skeleton, Lidar, and image data. For the AVE dataset, we adopt ResNet18 [12] as the encoder backbone and map the input data into a 512-dimensional vector. In terms of the audio modality, the data is converted into a spectrogram of size 257×1004. For the visual modality, we randomly select 4 frames from the video clips to construct the training dataset. For the CGM dataset, we design a model with 4 convolutional layers and 1 average pooling layer as the encoder. Hyperparameters. For the ADM dataset, the learning rate is 1e-3, and the batch size is 64. Regarding the USC dataset, the learning rate is 1e-6, and the batch size is 16. For the USC, MHAD, and Flash datasets, the learning rate is 1e-3, and the batch sizes are 16, 16, and 96, respectively. For the AVE dataset and the CGM dataset, we set the learning rates to 1e-4 and 1e-3 respectively and the batch size is set to 64. In this experiment, we employ the SGD optimizer [44] with a momentum of 0.9 and a weight decay of 1e-4. Baselines. To verify the proposed ERL-MR strategy, We adopt the following state-of-the-art "modality imbalance" solutions as the baselines: PMR [9] , MMCosine [68] , OGM-GE [40] and Gradient-Blending [60] . The complete baselines information is provided in the full version. To ensure fairness, we make efforts to reproduce all baseline methods and utilize the same network structure, number of training epochs, and SGD optimizer in all tasks.In this section, We have provided relevant experimental results on multiple research questions (RQ1-RQ6) related to the ERL-MR strategy on six multimodal datasets. Overall Performance of ERL-MR with Different Conventional Fusion Methods (RQ1). We first apply the ERL-MR to several conventional fusion methods, including Concatenation, Summation, FiLM [41] , and Gated [20] , and subsequently evaluate their performance on multiple datasets, as presented in Table 2 . The table also includes the performance results of each uni-modality. The results reveal significant variability in the performance of each uni-modal model across different datasets, indicating a common phenomenon of performance imbalance in multi-modal learning. It is noteworthy that in certain cases, the performance of the unimodal models outperformed that of the multi-modal models employing the standard fusion methods. This finding, exemplified by Sec. 3.1, clearly illustrates the inhibitory relationship among modalities. Furthermore, based on experimental results, we find that the performance impact of dataset size on the ERL-MR strategy is obvious. The smaller datasets inherently possess limited feature information, resulting in relatively minor small performance enhancements. Conversely, the larger datasets exhibit a more substantial improvement effect. For example, ERL-MR strategy has yielded remarkable results in the CGM dataset. The multi-modal model leveraging this strategy surpasses the performance of the top-performing uni-modal model and overcomes the suppressive effect between modalities. In general, the application of our ERL-MR strategy leads to significant improvements on four datasets in comparison to each fusion method.Performance Comparison with State-of-the-art Baselines (RQ2). We conduct a comparative analysis of our ERL-MR strategy against four state-of-the-art imbalanced modality modulation schemes: Gradient-Blending, OGM-GE, MMCosine, and PMR. For a comprehensive and fair comparison, in this experiment, we also test the performance of baselines using different fusion methods. As shown in Table 3 , the experimental results show that the baseline methods outperform conventional fusion methods, yet they are still significantly surpassed by our proposed ERL-MR strategy. The main factor contributing to this improvement is the design of our Euler Transformation, which enhances the representation capabilities of various modalities, enabling them to reinforce mutually. When combined with the MMCloss, the ERL-MR strategy achieves even more substantial performance gains, benefiting from as the effective constraints ensure the accurate model optimization. Furthermore, it is worth mentioning that Gradient-Blending necessitates the training of an additional uni-modal classifier and thus involves additional computations to validate the results. OGM-GE can only be directly applied in the form of concatenation and is not applicable to not in FiLM, thereby necessitating periodic modulation during the training process. PMR requires additional calculations for the prototype, resulting in significant computational overhead. In contrast, our method merely requires the implementation of a straightforward Euler component and is independent of both the fusion method and the network structure. This enables our approach to be applicable to a wider range of scenarios. Modality Imbalance Mitigation Performance Demonstration (RQ3). In this experiment, we demonstrate the improvements in modality balance performance achieved by the ERL-MR strategy.The experimental results are depcited in Fig. 2 . Specifically, Fig. 2a demonstrates the performance discrepancy between the visual and audio modalities in the AVE dataset. Without the applying ERL-MR strategy, the visual modality exhibited lower performance, while the audio modality demonstrated higher performance. With the applying ERL-MR strategy, however, we observed a continuous improvement in the performance of the visual modality during training, eventually reaching a performance level similar to that of the audio modality. In addition, we can find that the performance of the audio modality is further improved with the help of ERL-MR.Both modalities outperform the results obtained without ERL-MR, demonstrating a relatively balanced performance. This indicates that ERL-MR alleviates the modality imbalance and achieves the better uni-modal performance in this case. Likewise, the similar results are also observed on the MHAD dataset, providing additional support for our assertion, as illustrated in Fig. 2b . To explore the performance of ERL-MR in more than two modalities application scenarios, we conducted experiments on the Flash dataset. The results are shown in Fig. 2c . Upon applying ERL-MR, the performance enhancements across all modalities, albeit to a lesser extent in the GPS modality. We attribute this observation to two factors. First, the tasks involving more modalities (more than 2) tend to be more complex, resulting in leading to the increased task complexity and inter-modality connections. Second, GPS data often exhibits high randomness, making the information obtained less effective during training. Nonetheless, the results in Fig. 2c demonstrate the effectiveness of our ERL-MR strategy in balancing uni-modal performance, even in complex tasks involving multiple modalities.Performance under Diverse Real-world Tasks (RQ4). Autonomous Driving. In this experiment, we conduct experiments on the Flash dataset and present the experimental results in Fig. 3a . The results reveal a significant modality imbalance issue among the three modalities of the Flash dataset when ERL-MR strategy is not applied. In this scenario, the GPS and lidar modalities are underutilized, and the image modality dominates. In this context, the implementation of ERL-MR strategy not only enhances the performance of the fusion network but also leads to better performance of the lidar modality. Additionally, due to the inherent 5 , where E represents the Euler feature transformation method, C1 and C2 denote the cosine similarity constraint and cross-entropy constraint in MMCLoss, respectively. It is important to note that C1 and E are tightly coupled in our strategy, thus the separate performance experiments using only C1 were not conducted. The experimental results demonstrate that, except when only C2 is applied, using each component individually or in combination results in performance improvements compared to the baseline method. This indicates an essential relationship between improving model performance and the inclusion of component E. Additionally, combining component E with either C1 or C2 results in limited improvement compared to applying component E alone. However, when all three components are simultaneously applied, the performance is further improved, providing additional evidence of the effectiveness of each component in the ERL-MR strategy. Specifically, component E provides fine-grained enhancement features for each modality, while C1 and C2 guide the overall optimization direction by constraining these enhancement features. This strategy addresses the issue of modality imbalance and bolsters the performance of multi-modal models. 
conclusion: 
 In this work, we introduced the ERL-MR strategy as a novel approach to mitigate modality imbalance in multi-modal learning. This strategy effectively transforms the competitive dynamics between modalities, ensuring a stable feature optimization direction and promoting a mutually reinforcing collaboration among modalities. Importantly, this strategy is versatile, accommodating various multi-modal learning tasks with diverse data modalities and network architectures. Through comprehensive analysis and experimental validation across different datasets, encompassing various modalities, network structures, and application scenarios, the proposed strategy has consistently demonstrated its efficacy. ERL-MR emerges as a robust solution to address the challenges posed by modality imbalance in multi-modal learning, leading to significant improvements in model performance. These findings highlight the promising potential of the ERL-MR strategy for advancing robust multi-modal learning in real-world application scenarios. 
 
 Appendix:  


Decision: The study introduces a new approach to addressing modality imbalance in multi-modal learning by employing Euler’s formula to map features into a complex space, thus transforming competitive relationships into mutually reinforcing ones. Reviewers praised the detailed analysis, and robust experimental validation, demonstrating performance improvements over existing methods. Concerns were raised about the originality and detailed explanation of the Euler Feature Transformation, the choice of experimental baselines, and some clarity issues in the presentation. The authors' rebuttal to some extent addressed these concerns, providing necessary clarifications and additional experiments, which strengthened the overall understanding and robustness of the work. Given the responses, the reviewers reached a consensus on recommending acceptance. The authors are encouraged to revise the paper according to the reviewers' feedback to further enhance its clarity and quality.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose a Euler Representation Learning-based Modality Rebalance (ERL-MR) strategy, which reshapes the underlying competitive relationships between modalities into mutually reinforcing win-win situations. It is achieved by employing Euler’s formula to map features to complex space. The authors also introduce the multi-modal constrained loss based on cosine similarity to guide the optimization direction of the fusion network. By leveraging these approaches, the authors are able to solve the problem if modality imbalance, which impedes multi-modal learning from fully taking advantage of different modalities. Extensive experiments are conducted on several real-world datasets. And the paper is able to outperform the previous baselines.
strength_weakness: strength_weakness: + Well-written article with clear problem formation 
+ Detailed analysis and proofs are included.
+ Well-designed experiments and high performance over previous works. Ablation studies are also applied.
limitations: limitations: Weakness:
-Introducing existing solutions to the multi-modal area
-Less convincing baseline choices

Questions
I enjoyed reading this paper because it is easy for the reader to understand. The authors aim to solve the practical problem of modality imbalance in multi-modal computing. I also agree that the Euler feature representation method would be more effective. The experiments are well designed and convincing. However, I still have some questions about this paper.

The first question is about the Euler Feature Transformation. As introduced in Section 2.3, researches have been using Euler feature representations to capture the inherent geometric characteristics of the data. What is the major difference between this work and the previous works? The author claims that they are inspired by the Euler’s formulation. Do they simply migrate existing solutions to this new problem? Or the major contribution of this section is to build the complex mapping space?

The second question is about the experimental baselines. In Table 2, it seems that the authors directly use some simple models to serve as the baselines, i.e. Resnet 18 and simple 4-convolution, 1 average polling encoders. The multi-modal implementations of CGM are even far less effective than the uni-modals, which probably do not come from any articles. If SOTA baselines are applied, can this work still achieve similar performance? Similarly in Table 3, it would be more convincing if the authors could also compare the performance of their work and the SOTA models instead of only comparing with the SOTA “modality imbalance” solutions. 

Minor issues:
Some tables and plots are too small and hard to read, such as Table 1, Figure 2 and 3.
paper_summary: paper_summary: This paper proposes a novel strategy to address the issue of modality imbalance. The authors introduce the Euler Representation Learning-based Modality Rebalance (ERL-MR) strategy, which uses Euler’s formula to map features into a complex space, enhancing each modality and transforming competitive relationships into mutually reinforcing ones. Additionally, they propose a Multi-Modal Constrained (MMC) loss function that combines cosine similarity constraints of complex feature phases with cross-entropy loss to guide the optimization direction of the fusion network. Extensive experiments on six multi-modal datasets show that ERL-MR significantly outperforms state-of-the-art methods, achieving better modality balance and improved performance.
strength_weakness: strength_weakness: - This paper introduces a new approach by leveraging Euler’s formula to map features into a complex space, transforming competitive modality relationships into mutually reinforcing ones.
- Conducts extensive experiments on six real-world multi-modal datasets, demonstrating the practical applicability and effectiveness of the proposed method.
limitations: limitations: - The writing is quite poor, it is not clear enough to introduce EFT in detail, and how this relates to MMCLoss. E.g., the in eq. 4 what these two equations mean? How to learn the mapping? Besides, figure 1 is not clear, how the new features transform, and what is the meaning of the symbol after it. 
- The coverage of related work is insufficient. For instance, the use of Euler's formula to transform features is not a novel concept, and it is crucial to cite the original references and the starting points in ML and multi-modal.
- Given the description it is hard to reproduce the results, it will be better to have related codes.
- It is not clear to me if the EFT is a technique especially suitable for multimodal applications, or it can also work in a single modality.
- The transformation of features into a complex space might reduce the interpretability of the model’s internal workings, making it harder to understand how individual modalities contribute to the final decision-making process.
- The Multi-Modal Constrained (MMC) loss function, while theoretically sound, could introduce additional overhead in the training process, potentially leading to longer training times compared to simpler baseline methods. It will be better to have experiments to verify this.
paper_summary: paper_summary: This paper proposes an Euler Representation Learning-based Modality Rebalance (ERL-MR) strategy, which reshapes the underlying com- petitive relationships between modalities into mutually reinforcing win-win situations while maintaining stable feature optimization directions. Specifically,
strength_weakness: strength_weakness: The motivation is well-supported and reasonable.
The experiments is sufficient, including various datasets, tasks, and modalities
The idea to transform the vector feature representations of diverse modalities into a complex vector space is intritesting
limitations: limitations: It would be better to add more detail in Figure 1, such as the notation of encoders, and modality, which would help readers to understand the paper better.

The derivation of formula 5 is hard to understand.  Why are the real and imaginary parts expressed like this? what's the parameters u_j?

Why the sine and cosine operations inherent in Euler’s formula can alleviate feature differences between different modalities, transforming the competitive relationship between modalities into a cooperative one. It seems not intuitive

The reviewer concerns about the reliability of the experiments. For the AVE dataset, the performance is far lower than that in existing paper, including OGM, and PMR, with the same training parameters. Besides, other widely used datasets, like Kinects-sound, CREMA-D are not considered

In general, the reviewer appreciates the motivation of this paper,  the logic of the method writing and the reliability of the experiment do not convince me.
paper_summary: paper_summary: This work addresses the issue of modal imbalance in multi-modal learning and proposes a mutually reinforcing learning framework using Euler representation learning. Extensive experiments have demonstrated the effectiveness of this method.
strength_weakness: strength_weakness: ##  **This is an easy to understand work.**
1. The motivation is concise and reasonable, and the relevant work summary is substantial and rich.
2. The method statement is specific and appropriate, and theoretical analysis provides theoretical support for this work.
3. Abundant experiments have demonstrated the effectiveness of the proposed method.
limitations: limitations: ## **Some minor concerns.**
**This is a good job, and the method of presenting the entire text is natural and appropriate. I understand most of the content, and I think this is a good job, and I only have a few small confusions about the experimental part. If the author's clarification can help me understand these confusions, I am willing to improve my rating.**

**Some missing implementation details:** For fusion operations Concatenation, Summation, FiLM, and Gated in Table 2 and Table 3, the authors did not explain how they were integrated with the proposed framework, or in which part of Figure 1 these operations were inserted. For example, adding fusion operations after FC, according to my understanding. 

**Clarify some abnormal phenomena:** I understand that this work is similar to balancing the quality between different modalities. However, some counterintuitive results still require further clarification from the authors. For example, in terms of the reader's intuitive perception, after balancing like this work, the fusion effect should be even more outstanding due to the mutual promotion of learning effectiveness. However, on the CGM dataset, the fused effect does not seem to be better than the single-modal effect, at least as shown in Table 3. The effect of Colored image is much better than Fusion in most cases, and the authors need to explain this phenomenon. 

**Further experimental supplement on Flash and ADM datasets:** The authors should present a figure for ADM dataset, which is same as Flash dataset Figure 2(c). Moreover, The authors should also add a performance comparison among G-Blend, OGM-GE, MMCosine, and PMR methods on Flash and ADM datasets. Otherwise, this part of the content is to some extent missing. If this part of the experiment cannot be added, please also explain the reason.

**Prospects for future work:** It is appropriate to add some prospects for future work in the conclusion section to help readers grasp the improvement direction of this work in the future.