Enhance Multi-View Classification Through Multi-Scale Alignment and Expanded Boundary

Authors: Yuena Lin, Yiyuan Wang, Gengyu Lyu, Yongjian Deng, Haichun Cai, Huibin Lin, Haobo Wang, Zhen Yang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Multi-View Classification, Multi-Scale Alignment, Fuzzy Logic, Decision Boundary

Abstract:
Multi-view classification aims at unifying the data from multiple views to complementarily enhance the classification performance. Unfortunately, two major problems in multi-view data are damaging model performance. The first is feature heterogeneity, which makes it hard to fuse features from different views. Considering this, we introduce a multi-scale alignment module, including an instance-scale alignment module and a prototype-scale alignment module to mine the commonality from an inter-view perspective and an inter-class perspective respectively, jointly alleviating feature heterogeneity. The second is information redundancy which easily incurs ambiguous data to blur class boundaries and impair model generalization. Therefore, we propose a novel expanded boundary by extending the original class boundary with fuzzy set theory, which adaptively adjusts the boundary to fit ambiguous data. By integrating the expanded boundary into the prototype-scale alignment module, our model further tightens the produced representations and reduces boundary ambiguity. Additionally, compared with the original class boundary, the expanded boundary preserves more margins for classifying unseen data, which guarantees the model generalization. Extensive experiment results across various real-world datasets demonstrate the superiority of the proposed model against existing state-of-the-art methods.

Decision: accept (poster)
Reviewer scores: 5, 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1730201172159

thread_type:
official_review

<thread_object>
creator: reviewer_8qzm

<content>
summary:
This paper delves into the problems in multi-view data, namely fusion difficulty caused by feature heterogeneity and boundary ambiguity caused by information redundancy. To tackle these problems, this paper proposes a novel multi-view learning model MAMC, which reduces the fusion difficulty by introducing a multi-scale alignment strategy to mine the inter-view commonality and inter-class difference, while alleviating the boundary ambiguity by proposing an expanded boundary to learn a clear decision boundary. Overall, this model is interesting and works well to address the problems.

strengths:
- The paper structure is logical and the motivation is clear and easy to understand.

- The design of the expanded boundary is detailed, actually, it is interesting and seems simple-yet-effective.

- The authors provide the theoretical proof to explain the construction of the expanded boundary.

- The MAMC has very good performance in the classification results and visualization.

weaknesses:
- It would be better for the authors to provide the additional results with the general prototype in the ablation study, which makes it easier to understand the advantage of the expanded boundary. 

- The model is simple-yet-effective, however, I don’t know whether it is efficient. So the authors could provide a complexity analysis to understand the model efficiency better.

questions:
- Though the visualization is quite good, I am confused about how they obtained the results. They should explain how to combine the multi-view features to realize t-SNE visualization, such as concatenation or other ways.

- I find the comparative models in the paper mostly deal with uncertainty. Thus, I want to know whether the proposed model can be regarded as one robust model to handle uncertainty. And what are the differences between those models and MAMC?

- Whether the authors give the wrong symbols in Figure 4? As the caption described, the symbols in (c) and (d) should be two temperature coefficients, and the authors need to check it up.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730476526778

thread_type:
official_review

<thread_object>
creator: reviewer_9q5p

<content>
summary:
This papers addresses two issues in multi-view data: feature heterogeneity and information redundancy, by proposing a multi-view representation learning approach that enhances multi-view classification through multi-scale alignment and boundary expansion. Specifically, to address feature heterogeneity, this paper proposes a multi-scale alignment module that aligns from both instance and prototype scales, to explore commonalities from both inter-view and inter-class perspectives, thereby mitigating the issue of heterogeneity. To address information redundancy, this paper proposes a new expanded boundary method that adaptively adjusts boundaries using fuzzy set theory to handle fuzzy data. By integrating this expanded boundary with the prototype scale alignment module, the representation can be tightened and boundary ambiguity reduced, thereby enhancing the model's generalization capabilities. Experiments show that this model outperforms existing methods on multiple datasets.

strengths:
This article utilizes fuzzy sets to propose a novel self-adaptive expanded boundary method to enhance the separability of inter-class representations and the compactness of intra-class representations in multi-view data.

weaknesses:
1. The multi-scale alignment module proposed in this paper is an incremental approach. It is a commonly used paradigm in existing multi-view representation learning methods to enhance the discriminability and semantic consistency of representations.
2. The comparative methods in the experimental section are not comprehensive or representative enough. A thorough investigation of related work in the past three years should be conducted to demonstrate the effectiveness of this method.
3.One of the main problems addressed in this paper is feature heterogeneity, which is explained as the fusion difficulty caused by heterogeneity among multiple different data distributions. However, the proposed multi-scale alignment module does not provide an in-depth analysis of this issue or explain how it solves the problem.
4. The explanation of symbols in the paper is not clear enough, such as in Eq.(16). 
5. The experimental section lacks an analysis of model convergence.
6.The datasets have not been cited.

questions:
Please refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655154835

thread_type:
official_review

<thread_object>
creator: reviewer_1xbb

<content>
summary:
This paper proposes a multi-scale alignment module and an expanded boundary based on fuzzy set theory to address feature heterogeneity and information redundancy in multi-view classification. The alignment module enhances feature fusion, while the expanded boundary adapts to ambiguous data, reducing boundary ambiguity and improving generalization.

strengths:
1. The proposed MAMC model effectively addresses feature heterogeneity by employing view-specific auto-encoders and a multi-scale alignment module, which extracts common features across views, enhancing feature fusion and improving classification performance.
2. The model introduces an expanded boundary to manage information redundancy and class ambiguity, leveraging class differences to create clearer, more adaptive decision boundaries, which aids in distinguishing classes more effectively.

weaknesses:
1. The details of the paper need further refinement. For example, in subfigures (c) and (d) of Figure 4, the axis labels should use "temperature coefficients" instead of "penalty coefficients."
2. Equation 17 represents the overall optimization objective, involving four different loss functions. Why do the penalty coefficients apply to only two of these loss functions?
3. Figure 5 in the appendix should include variance, as it is a key aspect of stability analysis.
4. There are many minor issues that need refinement. For example, the first column in Table 1 should be labeled "datasets" instead of "methods."
5. The experiments were conducted using eight 24GB 3090 GPUs, but there are still out-of-memory issues reported in Table 1. This clearly indicates that the model's complexity and computational overhead require analysis and comparison, including metrics such as runtime, MACs, FLOPs, and parameters.

questions:
Please see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686854758

thread_type:
official_review

<thread_object>
creator: reviewer_p7j9

<content>
summary:
The authors propose an expanded boundary using fuzzy set theory to adaptively adjust class boundaries for ambiguous data, enhancing representation tightness and reducing boundary ambiguity. This expanded boundary also preserves more margin for unseen data, improving model generalization. Experiments across diverse datasets show the model’s superiority over state-of-the-art methods.

strengths:
1. The technique appears sound.

2. The presentation is clear and easy to follow.

3. The experimental results are sufficient to demonstrate the effectiveness of the proposed method.

weaknesses:
1. The model includes too many loss terms, which may limit its practicality for real-world applications, since it is often hard to set the hyperparameter to balance the contributions among them in practice. 

2. Figure 4 is unclear; I strongly suggest the authors present it in 2D for better readability.  Since there are too many dimensions making it hard to conduct the parameter sensitiveness analysis.

3. Some relevant references are missing [1][2][3], since [1] and [3] also study multi-view classification tasks and [2] focus on multi-view learning from a fuzzy view. The authors should discuss and compare with them. 
[1] Multi-view Classification Using Hybrid Fusion and Mutual Distillation, WACV, 2024.
[2] Self-Supervised Information Bottleneck for Deep Multi-View Subspace Clustering, TIP, 2023.
[3] Multi-View Fuzzy Classification With Subspace Clustering and Information Granules, TKDE, 2023.

questions:
Please address the question above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732768322900

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

Following the discussion phase, we have updated our manuscript to address the points raised in our earlier exchanges. To assist with your review, we have highlighted the revised sections directly in the document for your convenience.

We sincerely appreciate your time and thoughtful feedback. Please let us know if there is anything further we can clarify.

Best regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732966570718

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

Thank you again for your valuable feedback. We have carefully addressed your comments in the rebuttal and revised the manuscript accordingly.

We kindly invite you to review the updates, and please let us know if you have any further questions or suggestions. Your time and insights are greatly appreciated.

Best regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733105980184

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,
﻿

We hope this message finds you well. We have addressed your comments and revised the manuscript accordingly, submitting both the updated version and our rebuttal for your review.
﻿

With the rebuttal deadline approaching, we would greatly appreciate it if you could review the changes and provide your feedback. Please do not hesitate to reach out if there are any questions.
﻿

Thank you for your valuable insights and time.
﻿

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents the MAMC model to address feature heterogeneity in multi-view data through view-specific auto-encoders and a multi-scale alignment module. The paper is well-structured, with clear motivation, detailed design, and theoretical proof. Despite these merits, reviewers raised several major concerns regarding computational complexity and efficiency, lacking comparisons with recently published methods. Other minors include clarity of explanations for feature heterogeneity and symbol definitions. After the discussion period, the authors were able to implement all the comments in the revised manual, and AC was convinced by their efforts and new content and thus recommended accepting this paper.