A Progressive Skip Reasoning Fusion Method for Multi-Modal Classification

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 07:02:41

Abstract: In multi-modal classification tasks, a good fusion algorithm can effectively integrate and process multi-modal data, thereby significantly improving its performance. Researchers often focus on the design of complex fusion operators and have proposed numerous fusion operators, while paying less attention to the design of feature fusion usage, specifically how features should be fused to better facilitate multi-modal classification tasks. In this article, we propose a progressive skip reasoning fusion network (PSRFN) to make some attempts to address this issue. Firstly, unlike most existing multi-modal fusion methods that only use one fusion operator in a single stage to fuse all view features, PSRFN utilizes the progressive skip reasoning (PSR) block to fuse all views with a fusion operator at each layer. Specifically, each PSR block utilizes all view features and the fused features from the previous layer to jointly obtain the fused features for the current layer. Secondly, each PSR block utilizes a dual-weighted fusion strategy with learnable parameters to adaptively allocate weights during the fusion process. The first level of weighting assigns weights to each view feature, while the second level assigns weights to the fused features from the previous layer and the fused features obtained from the first level of weighting in the current layer. This strategy ensures that the PSR block can dynamically adjust the weights based on the actual contribution of features. Finally, to enable the model to fully utilize feature information from different levels for feature fusion, the skip connections are adopted between PSR blocks employing them. Extensive experiment results on six real multi-modal datasets show that a better usage for fusion operator is indeed able to improve performance.

Main paper: 
 related work 2.1 multi-modal classification: 
 Multi-modal classification (MMC) aims to achieve a better and robust classification performance by integrating multiple features. Formally, let A multi-modal classification learner can be denoted as a twotuple 𝔏 = (ℎ, F ), where ℎ is a learned decision function also called a classifier; F is a fusion function and it generally takes all view features and outputs their fused vector as the input of ℎ.X = R 𝑚 1 × R 𝑚 2 × • • • × R 𝑚 𝑃 denoteThe quality among multi-view features is different, lots of works consider their contribution to the final tasks. Multi-modal classification methods can be roughly divided into the feature level fusion-based (also namely early fusion) and decision level fusionbased (also namely late fusion). Based on where the view contribution are considered, these methods can be grouped into feature level weighting-based method (FW) and decision level weightingbased method (DW). FW learns the contribution weight of each feature [5, 15, 38, 41] . For example, EmbraceNet assigns 1 to the weight value of one view while 0 to others for each example according to a multinomial distribution. Yang et al. [38] proposed an adaptive-weighting discriminative regression approach (AWDR). AWDR adopts the square root form of view weight to distinguish features from different views. Zhang et al. [41] proposed discriminative multi-view fusion via adaptive regression (DMVF), it simultaneously discriminates the contribution diversity of different views and samples in an adaptive weighting manner, reducing the influence of low-quality views and outliers for classification. DW learns to assign weights at the decision level. For example, Han et al. [8] proposed a trusted multi-view classification (TMC), which models the confidence of each view at an evidence level using the Dempster-Shafer theory. Liu et al. [25] noticed that the fusion with Dempster's rule will produce counter-intuitive results, and adopted the opinion aggregation strategy to model the contribution degree of each view decision results.Fusion function F plays a very important role in our PSR block. We will review related multi-modal classification methods from the perspective of the adopted fusion ways, then select some of them to design PSR block.In the literatures of multi-modal learning, a common baseline for information fusion is to simply concatenate feature vectors across different modalities [32] such as images and texts, which is surprisingly performant in various multi-modal tasks. Other simple yet powerful methods include element-wise addition, multiplication and max-pooling [21] . Nonetheless, these element-wise operations are often limited in expressiveness as some noisy modalities could clutter the entire feature vectors. Concatenation: We combine the information from both modalities using concatenation, i.e.,EQUATIONwhere the function MLP is to map each view into a same size dimension, [•, •] is the concatenation operator. The drawback of this fusion strategy is still considering the multi-view information as separate as each of them affects the classification independently.Addition: We combine the information from both modalities using element-wise addition, i.e.,EQUATIONMultiplication: We combine the information from both modalities using element-wise product, i.e.,EQUATIONMax-pooling: We combine the information from both modalities using element-wise maximization, i.e.,EQUATIONwhere (•, •) denotes concatenate two vectors.According to the work [21] , the addition has more chance to achieve a better performance and does not introduce extra learnable parameters. Hence, it is selected the representative of the five basic fusion operators to introduce to our PSR block.Recent bilinear models consider all pairwise interactions among given view features, providing richer expressive capacity than linear models. Recently, this technique has been successfully applied into egocentric hand action recognition [29] , fine-grained visual recognition [23] and visual question answering (VQA) [17, 39] . In this section, we review two respective effective and efficient implementations of bilinear pooling.Given two feature vectors 𝑣 1 and 𝑣 2 from two views, they can be fused with bilinear pooling as follows:EQUATIONwhere 𝑊 𝑖 ∈ R 𝑚×𝑛 is a projection matrix, 𝑐 𝑖 ∈ R is the output of the bilinear model, and 𝑏 𝑖 is a bias for the output 𝑐 𝑖 .In the pooling processing, the bilinear method will yield a very high-dimensional fused vector and bring parameters of 𝐿 × (𝑁 × 𝑀 + 1) including a bias vector 𝑏, where 𝐿 is the number of output features. It is not friendly fusion way for various computational resources. To overcome the problem of the curse of the parameters, Kim et al. proposed [17] the multi-view low-rank bilinear pooling (MLB) approach. First two input vectors (i.e., the image feature 𝑥 ∈ R 𝑚 and the text feature 𝑦 ∈ R 𝑛 ) are embedded into same dimension space with two low-rank projection matrices using two linear mappings without biases. Then, a joint representation is learnt with Hadamard product (element-wise multiplication) in a multiplicative way. Finally, a linear mapping with a bias is used to project the joint representations into an output vector for a given output dimension. To further increase model capacity, nonlinear activation like tanh is added after 𝑐. Since the MLB approach can generate feature vectors with low dimensions and deep networks with fewer model parameters, it has achieved very comparable performance to MCB. The process can be formalized asEQUATION) As pointed out by Yu et al. in [39] , with the same dimensionality for the output features, MLB may suffer from insufficient representation due to approximating the outer product using a simple way that the features are directly projected to the low-dimensional output space and element-wise multiplication is performed. To overcome this problem, Yu et al. [39] proposed multimodal factorized bilinear pooling (MFB). First, the features from different modalities are expanded to a high-dimensional space and then integrated with Hadamard product. After that, sum pooling followed by the normalization layers are performed to squeeze the high-dimensional feature into the compact output feature.EQUATIONwhere the function SunPool(𝑥, 𝑘) means using a one-dimensional non-overlapped window with the size 𝑘 to perform sum pooling over 𝑥.It is clear from the formulas ( 5)-( 7) that the key to advanced fusion operators lies in the outer product. Hence, the outer product is introduced to our PSR block. 
progressive skip reasoning fusion network: 
 In this section, we first introduce a progressive skip reasoning (PSR) block, and then the progressive skip reasoning fusion network (PSRFN) for multi-modal classification is desinged by stacking multiple PSR blocks.As shown in Fig. 1 The 𝑣 𝑙 𝑖 only depends on 𝑣 𝑙 -1and can be obtained by the following mapping𝑣 𝑙 𝑖 = 𝑀 𝑙 𝑖 𝑣 𝑙 -1 𝑖 (8)where 𝑀 𝑙 𝑖 is a mapping function for extracting more high-level feature representation 𝑣 𝑙 𝑖 . This mapping function can be easily performed using a fully-connected layer followed by ReLU activation function.Feature fusion: Fusion plays a very important role in multimodal classification task. The PSR block is armed with two fusion operators. One is to fuse multiple view representation, and the other is to fuse the current fused feature representation and that of previous layer. It is well known that the quality of views usually differs. Similarly, the contribution of both of the previous fusion vector and current fusion vector may be different to final task. Based on above analysis, feature fusion is performed by the dualweight fusion strategy, which takes |𝑉 | transformed view feature vectors and the fusion vector from the previous layer, and outputs the fusion vector for the current layer. Specifically, this process of feature fusion can be divided into two steps for execution.Firstly, it assigns weights to |𝑉 | transformed view feature vectors based on their contribution to the classification task, and then cascades and transforms the weighted features into the fused transformed view feature. The cascading operation is able to not only unify multiple views together but also preserve their characters.In our experiment, the weighted cascading function 𝑓 𝑙 𝑣 is designed by weighted concat of |𝑉 | transformed view feature vectors𝑣 𝑙 1 , 𝑣 𝑙 2 , • • • , 𝑣 𝑙 |𝑉 |at the 𝑙-th layer, and can be obtained by following mappingEQUATION)where ⊕ is a cascading operator, 𝑤 𝑙 𝑖 is the automatically tuned weight parameter of 𝑖-th view at the 𝑙-th layer.Next, the fused transformed view feature 𝑣 𝑙 can be obtained by following mappingEQUATION)where 𝑀 𝑙 𝑣 is a mapping function transforms feature to the same dimension as the 𝑣 𝑙 𝑖 , and it is performed using a fully-connected layer followed by ReLU activation function.Secondly, it further adjusts the weights to the fused transformed view feature from the current layer and the fusion vector from the previous layer based on their contribution to the classification task, and then transforms the weighted features into the fused feature for the current layer. Specifically, let 𝑐 𝑙 -1 be the fusion vector at the (𝑙-1)-th layer, and the 𝑐 𝑙 𝑚 is transformed from 𝑐 𝑙 and can be obtained by following mapping.EQUATIONwhere 𝑀 𝑙 𝑚 is a mapping function for extracting more high-level feature representation 𝑐 𝑙 𝑚 , just like 𝑀 𝑙 𝑖 . The fusion function 𝐹 𝑙 is used to weighted fuse 𝑣 𝑙 and 𝑐 𝑙 𝑚 . In our experiment, the fusion operator is designed as element-wise addition or outer product, and it can be chosen one of them. The element-wise addition fusion operator is able to enhance every feature used for fusion, thereby strengthening the fused feature. The outer product operator can expand the dimension of the input vector to a higherdimensional space, thus providing enhanced feature representation capabilities. Afterwards, the output of fusion function 𝐹 𝑙 is mapped through a mapping function to produce the final fused vector 𝑐 𝑙 . This step is implemented as follow:EQUATIONwhere 𝑤 𝑙 𝑓 𝑣 and 𝑤 𝑙 𝑓 𝑚 are the two automatically tuned scale parameters, the ⊗ is the element-wise addition or outer product fusion operator, 𝑀 𝑙 𝑐 is a mapping function just like 𝑀 𝑙 𝑖 too. Since element-wise addition and outer product in Eq. 12 are introduced, the magnitude of the output neurons 𝑐 𝑙 may vary dramatically, and the model might converge to an unsatisfactory local minimum. In this paper, we use normalization [13] to address this problem. Specifically, we use the power normalization and ℓ 2 normalization [7, 39] . This process is implemented as follows:EQUATION)EQUATIONCompared to most existing fusion strategies that only fuse multiple view representation once via single fusion operator, the proposed PSR block can deal with two types of information stream: view representation and fused feature representation; Moreover, PSR can be simultaneously armed with two fusion operators, so it has greater flexibility for using exiting fusion operators. This reliefs the sub-optimal issue due to insufficient interaction among views.We propose the progressive skip reasoning fusion network (PSRFN) by stacking multiple PSR blocks shown in Skip: The skip connection approach is adopted among PSR blocks, it is implemented by directly connecting the output of a PSR block to the input of subsequent PSR blocks, enabling efficient information flow between blocks. Specially, the output 𝑐 1 of first PSR block serves as the part of the input for the 2-th to the 𝐿-th PSR blocks, the output 𝑐 2 of sencond PSR block serves as the part of the input for the 3-th to the 𝐿-th PSR blocks, and so on in this manner.Skip connections establish direct information channels between different layers, enabling the fusion of features from different layers, allowing the network to learn richer feature representations and more effectively utilize the feature information from each block during training. This contributes to improving the PSRFN's performance, convergence speed, and generalization ability in completing multi-modal classification tasks.Loss: Ensemble learning, as an effective technique to integrate multiple models, tells us that the accuracy of each model is important to the final result [42] . Inspired this, different from most where 𝑆 𝑗 is used to map 𝑣 𝐿 𝑗 into a special space whose dimensional is the same as the number of classes, and then the mapped vector is transferred a probability vector ŷ𝑗 with the Softmax function.The total losses are defined asEQUATION)where L 𝑆 is to ensure the accuracy of model taking each view and it is defined asL 𝑆 = 1 |𝑉 | |𝑉 | ∑︁ 𝑗=1 L 𝑗 𝑆 = - 1 𝑛|𝑉 | |𝑉 | ∑︁ 𝑗=1 ∑︁ (𝑥,𝑦) ∈𝐷 𝑦 log( ŷ𝑗 )L 𝐹 is to ensure the accuracy of the fusion features and it is defined asL 𝐹 = - 1 𝑛 ∑︁ (𝑥,𝑦) ∈𝐷 𝑦 log( ŷ𝑐 )𝜇 and 𝜆 are nonnegative tradeoff parameters controlling the relative contributions of the corresponding loss terms. The flowchart of the entire PSRFN is illustrated in Fig. 2 .It should be noted that (1) the fusion operator used in this article can be replaced with other fusion operators such as bilinear pooling and tensor-based fusion methods; (2) early fusion, intermediate fusion and late fusion are three special cases of PSRFN. 
experimental studies: 
 In the experiments, we evaluate the effectiveness of the proposed PSRFN algorithm on six multi-modal classification datasets. Our computational environment is Ubuntu1 16.04.4, 512 GB DDR4 RDIMM, 2X 40-Core Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GH and NVIDIA Tesla P100 with 16GB GPU memory.Our experiments are conducted on six challenging multi-modal classification datasets which include image, text, audio, depth and video datasets. (1) Animals with Attributes (AWA) [18] dataset, which includes 30,475 images from 50 categories with seven view features. These seven views consist of six low-level features and one deep feature. (2) NUS-WIDE-128 (NUS) [30] dataset, which includes 43,800 samples from 128 categories with seven view features. These seven views consist of six image features and one text feature. We select a subset of 10 categories from this dataset, with a total of 23,438 images. (3) Reuters [1] dataset, which includes 111,740 samples from six categories with five multilingual view features. To enable the model to process this dataset, PCA is used to reduce the dimensions of all views to 1000. According to [8, 24] , Gaussian noise is added to either 5-view or 3-view datasets, resulting in two versions named Reuters5 and Reuters3, respectively. (4) VoxCeleb [27] dataset, which includes 153,516 samples from 1,251 categories with five audio view features. These five views consist of three traditional features and two deep features. To achieve the research objective, Gaussian noise is added to the two deep features.(5) YoutubeFace dataset, which includes 3,425 videos from 1,595 different people with five view features. According to [33] , we use a subset of 31 categories from this dataset, with a total of 101,499 frames.We employ five measures to evaluate the performance of each method, which are accuracy, recall, precision, F1 score and kappa, respectively. Accuracy, recall and precision are widely-used measures. F1 is an indicator in statistics that comprehensively assesses the performance of the model by calculating the harmonic mean of precision and recall. Wang et al. [31] pointed that the accuracy measure cannot recognize the random consistency and may cause an unreliable evaluation. Hence, one more reasonable measure: kappa is also used. These five evaluation metric definitions are as follows:EQUATIONEQUATIONwhere 𝑛 denote the number of all samples; 𝑇 𝑃 and 𝑇 𝑁 denote the number of true positive and true negatives, respectively; 𝐹 𝑃 and 𝐹 𝑁 denote the number of false positives and false negatives, respectively. 𝑝 𝑜 = 𝑇 𝑃 +𝑇 𝑁 𝑛 is the empirical probability of agreement on the label assigned to any sample (the observed agreement ratio), and𝑝 𝑒 = (𝑇 𝑃 +𝐹 𝑁 ) × (𝑇 𝑃 +𝐹 𝑃 ) 𝑛 + (𝐹 𝑃 +𝑇 𝑁 ) × (𝐹 𝑁 +𝑇 𝑁 ) 𝑛is the expected agreement when both annotators assign labels randomly.The larger values of the five metrics indicate a better classification performance. For each compared algorithm, 5-fold crossvalidation is performed on each dataset, and the mean metric value and the standard deviation are recorded for comparative studies.PSRFN is a weighting-based multi-modal classification method (MMC). To validate the effectiveness of the proposed PSRFN, comprehensive comparison experiments are conduced with eight related weighting-based MMC methods. The compared methods can be classified into the following two groups according to the level of weighting:(1) The first category is the feature level including EmbraceNet [5] , AWDR [38] and RAMC [15] . EmbraceNet assigns 1 to the weight value of one view while 0 to others for each example according to a multinomial distribution. AWDR is an adaptive-weighting discriminative regression approach. Following [38] , the parameter 𝜆 is chosen from the set {10 -3 , 10 -2 , • • • , 10 3 }, while 𝑘 varies within the range {1, 3, • • • , 9}. RAMC employs an 𝐿 2,1 -norm loss function to acquire a joint weighted projection space across all views. This method preserves the correlation and diversity among views through a selfsupervised weighting strategy. Similarly, the parameter 𝜆 is chosen from {10 -3 , 10 -2 , • • • , 10 3 }, and 𝑘 ranges from{1, 3, • • • , 9}. (2)The second category is the decision level including BV [21] ,SSV [21] , MR [21] , TMC [8] and TMOA [25] . BV assigns 1 to the weight value of the view with the best performance while 0 to others according to whole classification performance of each view. MR assigns 1 to the weight value of the view with the best performance while 0 to others for each example according to the classification performance of each view of each example. SSV assigns the same values to all views. TMC and TMOA are two well-known trusted fusion methods. Both of them model the confidence of each view at an evidence level with the Dempster-Shafer theory and opinion aggregation.For the proposed PSRFN, in order to make the model more elegant and lightweight, we set each module to contain only one fully connected layer, and the number of neurons in the fully connected layer is selected from [64, 128, 256] . In light of comparison to other models, 𝜇 and 𝜆 are fixed 1. PSRFN (add.) and PSRFN (op.) represent the use of element-wise addition fusion operator and outer product fusion operator, respectively, within the fusion function 𝐹 𝑙 .The results, displayed in Tables 1 and 2 , are presented through the mean metric value and the standard deviation obtained from 5-fold cross-validation. Avg. represents the average performance value of each algorithm across the six datasets, while Rank indicates the average ranking of each algorithm among all compared methods across the same datasets. From Tables 1 and 2 , the following observations can be made:(1) Compared to the BV method that can be viewed as single view method, PSRFN achieves the best classification performance, while other multi-modal fusion methods perform worse than BV on some datasets in terms of accuracy, recall, precision, F1 and kappa. For example, the accuracy of EmbraceNet and TMC is lower 1.11% and 10.83% than BV, respectively, while the proposed PSRFN is higher by 4.1% than BV on YoutubeFace dataset. The reason may be that PSRFN progressively enhances the fused features in a multiple interaction manner through adaptive combination with exiting fusion operators; while there is once interaction and no interaction for the multi-view features in EmbraceNet and TMC, respectively. This shows the advantage of our progressive fusion strategy. In summary, the above evidence shows the design rationality of the PSRFN. Hence, it is worthwhile to pay more attention to the fusion operator usage and fused feature enhancement. 
conclusion: 
 In this paper, we have proposed a progressive skip reasoning fusion network (PSRFN). Unlike most existing multi-modal fusion methods that only use one fusion operator in a single stage to fuse all view features, PSRFN utilizes a PSR block to fuse all views with the fusion operator at each layer. The comprehensive experimental results have verified the effectiveness of the proposed method, suggesting that the multiple interaction between fused feature and view features is beneficial. In the future, it is worthwhile to propose better progressive fusion strategy. It has been observed that PSRFN needs to be armed with different configurations. Hence, it is interesting to apply neural architecture search technique to automatically search for the proper configurations.  
: 
 view (it is noted that like other works [9, 22] the terms "view" and "modality" are used interchangeably in this paper). For example, we can extract color, shape and texture features from images, multiple kinds of language features from texts, and acoustics and energy features from audio. Multi-modal data have facilitated the development of many tasks such as classification [16, 20, 34] , clustering [35, 36, 40] , and feature selection [6, 14] .Among these multi-modal tasks, multi-modal classification technique attracts more and more attention of researchers from academics and industry with increasing demands for multimedia data management. Many studies show an effective fusion operator is very important for such task [3] . Hence the researchers in this community pay enormous attention to the fusion operator design. For example, Liang et al. [22] introduced the association information between modality features into multi-modal data fusion and proposed an association-based fusion strategy for multi-modal classification (MMC) in an interpretable manner. Hu et al. [11] proposed a fuzzy fusion method and this method used a fuzzy operator to fuse all kernel matrixes that are obtained by operating different kernel functions on each view. Liu et al. [26] developed an efficient low-rank multi-modal fusion method with modality-specific factors to address the problem of tensor-based methods suffering from exponential increase in dimensions and in computational complexity. Hou et al. [10] aimed to address the problem of existing tensorbased methods ignoring the complex local intercorrelations and proposed a polynomial tensor pooling method (PTP) for integrating multi-view features by considering high-order moment. So far, a huge number of fusion operators [2, 29, 37] have been proposed. However, fusion operator usage is rarely studied.Moreover, most existing fusion methods only conduct once fusion for learned multi-view representation, and then the fused features are passed to a classifier for decision-making. The fused features obtained with the strategy may be sub-optimal due to insufficient interaction among views. One may ask one question that "Can the fused features be enhanced via multiple interactions with multi-view representation"?Many applications such as face recognition and speaker recognition benefit from the hierarchical features of deep learning. This suggests that the original features could not be the best fit for the current task and one or more feature transformations (namely representation learning [4] ) are needed for better performance. Similarly, hierarchical features for multi-view features are also being learned for performance gain. In this case, multi-view fusion faces one issue: which hierarchical features should the fusion operator operate on ? Moreover, in representation learning, to learn an effective feature representation, we always transform features using a layer-by-layer manner. Based on these observations, it is reasonable that fusion should occur on each layer and the fusion representation should be passed layer by layer.Accordingly, the article studies the issue of the fusion operator usage in hierarchical multi-view features. Specially, we propose a progressive skip reasoning fusion network (PSRFN). In PSRFN, views are transformed once, fusion will occur once. This process is performed by stacking multiple PSR blocks, and each PSR block is able to be armed with existing fusion operators. The PSR block takes view feature vectors at the current layer and the fused feature vector at the previous layer as inputs, fuse them, and then output the fused feature vector of the current layer.Our main contributions are summarized as follows:(1) A progressive fusion strategy is proposed, which incorporates the fused feature from the previous layer and the different view features from the current layeras the input for the current fusion layer, thus achieving progressive fusion. This strategy fully utilizes the complementarity of information from all views and gradually improves the discriminative ability of features. With the strategy, a progressive skip reasoning (PSR) block is proposed, which is easy to be modified and enhanced by replacing its fusion operation with existing fusion operators according to different tasks or datasets. (2) A dual-weighted fusion strategy with learnable parameters is proposed, which can automatically adjust weights based on the actual contribution of each view feature and fusion layer feature. This strategy divides the weighted fusion process into two steps to achieve precise control of weights and fully utilize information from all views. (3) A progressive skip reasoning fusion network (PSRFN) for multi-modal classification is designed by stacking multiple PSR Blocks. PSR is able to fuse different view features more than times according to the number of PSR blocks given by users. This way avoids the problem that users have to determine where views should be fused. Moreover, the information transmission between PSR blocks adopts skip connections, enabling the model to fully capture both deep and shallow features in multi-modal data, thereby enhancing the model's ability to understand the data. Additionally, skip connections also help alleviate the problem of gradient vanishing, allowing the model to learn feature representations more effectively during the training process. (4) Extensive experiments conducted on six real multi-modal datasets verify the effectiveness of PSRFN. 
 
 Appendix:  


Decision: A well-written paper on skip reasoning. Technically strong as well.

Reviews: review: paper_summary: paper_summary: This paper proposes a progressive skip reasoning fusion network (PSRFN) for multi-modal classification. Unlike most existing multi-modal fusion methods that only use one fusion operator in a single stage to fuse all view features, PSRFN utilizes a PSR block to fuse all views with the fusion operator at each layer. The comprehensive experimental results have verified the effectiveness of the proposed method.
strength_weakness: strength_weakness: This paper proposes a progressive skip reasoning fusion network (PSRFN) for multi-modal classification. This design method is novel and intersting.
limitations: limitations: Although this method is interesting, I still have some concerns that the author needs to explain:
1) Some detailed issues. All formulas have no punctuation marks, and through your symbols, it is not clear which are scalars, which are vectors, and which are tensors. Please rewrite.
2) Can your design idea be used in visual-language multi-modal classification tasks, such as multi-modal fake news detection, multi-modal sentiment analysis, multi-modal sarcasm detection, etc.? Please add some experiments to confirm your method.
3) Some symbols and letters in the frame diagram are not explained and I feel they are not rigorous enough. Please add them again.
4) How do you solve the problem of modal variability? If features conflict between multiple modalities, how does your design resolve them?
5) Your fusion seems quite simple, balanced through weight strategy, but I would like to ask how to explain the feature weight? You did not give some feature map visualization results in the experimental part. I want to intuitively see the feature visualization and fusion feature visualization of each single mode in the feature flow process, please add.
If the above issues are resolved, I will consider accepting it.
paper_summary: paper_summary: Authors have found that there is limited attention paid to the design of feature fusion usage in multimodal fusion studies. Therefore, the authors conducted research on how to better fuse features to enhance multimodal classification tasks. This paper proposed a progressive skip reasoning fusion network (PSRFN), which utilizes the progressive skip reasoning block to fuse all views with a fusion operator at each layer. This design enables the network to deeply understand and utilize multimodal information, leading to better performance in classification tasks. The PSRFN network provides novel ideas and methods for multimodal fusion research, demonstrating a certain degree of innovation and practicality.
strength_weakness: strength_weakness: This paper proposes a novel progressive fusion strategy that leverages the complementarity of multi-view information to gradually enhance the discriminative power of features. It also introduces a dual-weighted fusion strategy that precisely adjusts weights based on the actual contribution of each view feature and fusion layer feature, further improving the fusion effect. A progressive skip reasoning fusion network (PSRFN) for multi-modal classification is designed, eliminating the need for users to determine where the views should be fused. The network utilizes skip connections to fully capture both deep and shallow features, enhancing the model's ability to understand the data. Through extensive experimental validation, the authors proved that the design of feature fusion usage can indeed enhance performance, providing a direction for future research. The proposed PSRFN possesses a clear theoretical foundation, which reasonably explains the working principles and advantages of the model. It can also be extended to other multi-modal learning tasks, demonstrating broad application prospects.
limitations: limitations: 1.The abstract should be modified according to the "motivation, description, results, and conclusion" sections. I suggest expanding the conclusion section to emphasize the significance of the findings.
2.The content in section 4.2 is relatively cumbersome. It needs to be reorganized to make it more concise and comprehensible.
3.The description of the model training process is unclear. It is necessary to explicitly state the stopping criteria for training, such as performance on a validation set, the number of iterations, or other specific conditions.
paper_summary: paper_summary: A multi-modal fusion method is proposed based on the task of multi-modal learning, which combines feature-level and decision-level fusion.
strength_weakness: strength_weakness: ##  **A simple and effective method.**
1. The idea is not complicated, and readers in the relevant field can easily understand it.
2. The idea of progressive skip reasoning fusion is interesting. 
3. The motivation behind the method is clear, and the experiment is set based on the motivation, which complements each other.
limitations: limitations: ## **Some minor concerns.**
This work provides readers with a simple and effective method, which currently appears to be a good work. But I still have some suggestions for the authors to improve the quality of this paper.

**In the related work section:** The fusion methods in Subsection 2.3 may not be fully summarized. Because there are other multi-modal fusion methods, such as weighted averaging [1], self-weighting [2], attention [3], and so on. If we can increase the discussion of these works, it will help improve the clarity of the contribution in this article.

[1] Tensorized multi-view subspace representation learning. IJCV, 2020.

[2] Dynamic auto-weighted multi-view co-clustering. PR 2020.

[3] GAF-Net: Graph attention fusion network for multi-view semi-supervised classification. ESWA 2023.

**In the method section:** The authors are suggested to add an algorithm flow chart in Subsection 3.2, which enhances the summarization of this work. Then, a complexity analysis of the model should be also added.

**In the experiment section:** The experiment was slightly incomplete due to the lack of sensitivity analysis for parameters $\mu$ and $\lambda$.

**Other details:** The overall layout and standardization still need to be improved for clear and good paper quality. For example, i) punctuation is required after formulas; ii) Some expressions need to be unified, such as Fig. 2 or Figure 2 in the main body. 

**In summary, this is an easy to understand and effective multi-modal fusion work. Motivation, methods, and experiments are sufficient, but certain aspects can still be improved. If the authors are willing to adopt my suggestions to enhance the overall presentation quality of the paper, I am willing to increase my rating.**