Beyond the Known: Ambiguity-Aware Multi-view Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 17:00:45

Abstract: The inherent variability and unpredictability in open multi-view learning scenarios infuse considerable ambiguity into the learning and decision-making processes of predictors. This demands that predictors not only recognize familiar patterns but also adaptively interpret unknown ones out of training scope. To address this challenge, we propose an Ambiguity-Aware Multi-view Learning Framework, which integrates four synergistic modules into an end-to-end framework to achieve generalizability and reliability beyond the known. By introducing the mixed samples to broaden the learning sample space, accompanied by corresponding soft labels to encapsulate their inherent uncertainty, the proposed method adapts to the distribution of potentially unknown samples in advance. Furthermore, an instance-level sparse inference is implemented to learn sparse approximated points in the multiple view embedding space, and individual view representations are gated by view-level confidence mappings. Finally, a multi-view consistent representation is obtained by dynamically assigning weights based on the degree of cluster-level dispersion. Extensive experiments demonstrate that our approach is effective and stable compared with other state-of-the-art methods in open-world recognition situations.

Main paper: 
 introduction: 
 Different feature extractors or sensors can perceive various patterns of information conveyed by real-world objects. Multi-view data provides a wealth of information that can significantly enhance learning and understanding of models in analysis tasks [7, 26] . Therefore, multi-view learning can improve the robustness and reliability of recognition performance by integrating multiple perspectives or patterns of data as compared to single-pattern information processing. Indeed, due to constraints imposed by device limitations or working conditions, it is often expensive or even infeasible to gather comprehensive multi-view data encompassing all categories during the training phase [13, 20] . Although multi-view learning methods are well-developed, most work has focused on improving performance within known constraints. Nonetheless, the inference process in real-world testing environments is plagued by ambiguity, an aspect frequently neglected in previous works. Ambiguity essentially captures the gap between the controlled conditions of model training and uncontrollable factors encountered when the model is deployed in real-world scenarios. To encapsulate, the presence of ambiguity in scenarios of open multi-view learning remains a paramount obstacle, originating from the inherent variability and unpredictability of external factors.This ambiguity in multi-view data arises from its inherent variability, presenting a persistent issue for multi-view learning: how to reconcile disparate views to distill essential patterns amidst the noise and redundancy. Variability of multi-view data introduces consistency and complementarity nature that contribute to favorable inference performance [27, 29] , while additional messages that tend to disrupt decision-making are also conflated. Each view may capture distinct facets of the data, leading to varied representations that necessitate reconciliation for a comprehensive comprehension. A primary concern is the redundancy brought by an excessive amount of information, which includes irrelevant details that can obscure the essential patterns critical for precise classification. Concurrently, the disparity in feature spaces across different views, known as the heterogeneity gap, complicates the amalgamation of data samples from these disparate sources [12] . To tackle these challenges, recent studies have concentrated on developing a unified representation space that integrates intrinsic information from various views, thus facilitating a more coherent interpretation of multi-view data. Additionally, by learning to identify and prioritize the most task-relevant features across views, it is possible to significantly mitigate the negative influence of redundancy.Whereas external ambiguity introduced by unknown classes blurs the boundaries between categories in the representation space, thereby presenting issue: How to inform multi-view classifiers about the existence of its unknown classes so that it recognizes them with low confidence. Moreover, the open-world problem introduces safety risks, particularly when classifiers erroneously assign high confidence levels to unknown classes, potentially compromising decision performance [5] . The ability to simultaneously recognize known categories and reject unknown categories is the task of open-set recognition approaches [11, 18, 28] . Unfortunately, these methods predominantly focus on analyzing single-view feature data, which is prone to external disturbances, potentially leading to misrecognition. In contrast, multi-view learning leverages the complementary nature of diverse views to enhance the efficacy of open-set recognition. In order to avoid the problem of ambiguous category boundaries caused by insufficient categories of training multi-view data, we try to extract category information from the existing raw data for the model to adapt to the distribution of potentially unknown categories in advance. This preparation allows the model to adjust its recognition boundaries more dynamically, enhancing its ability to differentiate between known and unknown categories effectively.𝐗′ (2) 𝐗 (2) Randomized Perturbation 𝐗′ (1) 𝐗 (1) Figure 1 : Overall of the proposed Ambiguity-Aware Multi-view Learning Framework. The soft-mixup augmentation module generates mixing samples from the original ones, feeding them into the subsequent network alongside the original. The instance-level sparse inference module computes view-specific representations, while the view-level confidence network module calculates confidence maps for gating. Ultimately, a consistent representation is fused using inter-class dispersion weights via the centroid-guided adaptive fusion module. The entire framework is optimized with three losses.To jointly address the above-mentioned issues, in this work, we focus on generalization in open multi-view environments, aiming at exploring data distributions and underlying properties from observational data to enhance the ability to recognize potentially unknown classes. Specifically, we devise four distinct modules, each geared towards achieving specific objectives. The soft-mixup augmentation module is designed to generate soft-labelled synthetic instances in order to better manage open space risk. While the instance-level sparse inference module based on interpretable optimization objectives is applied to filter out the influence of redundant features and projects multi-view features into common representation space. Additionally, the view-level confidence network module estimates prediction confidence to gating view-specific representations. Finally, the centroid-guided adaptive fusion module dynamically modulates the contributions of different views based on inter-class dispersion, guaranteeing that the intrinsic information of multiple views is encoded into the multi-view consistent representation. The overall framework is illustrated in Fig. 1 . The main contributions of this paper can be listed as follow:• We propose Ambiguity-Aware Multi-view Learning Framework (AAML), which addresses the variability and unpredictability in open multi-view learning environments. • We employ mixed interpolated samples to occupy the unknown ambiguous space between categories to broaden the learnable feature space, in which the soft labeling strategy measures the uncertainty.• Extensive comparative experiments prove that the proposed AAML achieves stable correct classification rates at varying levels of false positive rate, highlighting its robustness and reliability in open-set classification tasks. 
related work 2.1 multi-view learning: 
 Existing multi-view learning methods are usually based on the assumption that the different feature views are projected from the potential feature space. Assuming that the training multi-view dataset D 𝑡𝑟𝑎𝑖𝑛 = {{X (𝑣) } 𝑉 𝑣=1 , y * } consists of 𝑁 instances across 𝐾 known categories with 𝑉 views, whereX (𝑣) = {x (𝑣) 𝑖 } 𝑁 𝑖=1 ∈ R 𝑁 ×𝐷 (𝑣)and corresponding true labelsy * = {𝑦 * 𝑖 } 𝑁 𝑖=1 with 𝑦 * ∈ Y = [1, 𝐾].In a multi-view scenario, the first step is to construct view-specific mappings to project various feature spaces X (𝑣) to uniform space H , where 𝑚 (𝑣) : X (𝑣) → H , where 𝑑 (𝑣) represents the dimensional of original features in 𝑣-th view.Such approaches adhere to a specific-uniform procedure where potential features are first extracted from each view and then combined to create a uniform representation. The main goal is to take advantage of the complementary and consistent information provided by multiple views, and to reduce the interference of invalid information such as redundancy and noise. To avoid degrading the quality of the derived representation, Wang et al. [19] proposed MetaViewer to guide the learning of uniform representations, formulating the extraction and fusion of view-specific features as a nested optimization problem. Similarly, Tang et al. [17] proposed deep network to achieve multi-view safeness by automatically selecting features while extracting complementary information and eliminating irrelevant noise. In order to keep the views with clear clustering structures from receiving constraints from views with ambiguous structures, Chen et al. [2] utilized global discriminative information to guide the learning of local common representation. Tang et al. [16] projected each view into a label space with consensus part and view-specific part, while cross-view similarity graph learning term is embedded to preserve the local structure. Liang et al. [14] factorized task-relevant information in multi-view data into shared and unique information, and removed task-irrelevant information via upper bounds on mutual information. Despite the increasing interest in multi-view learning methods, rarely has research focused on tailoring these approaches to open settings.Open-set recognition addresses confronts the challenge of correctly classifying samples from known categories while accurately identifying those from unknown categories. The main challenge is that incomplete knowledge exists in the training phase and potential unknown categories can be encountered in an algorithm during testing [6, 24] . Following predictive distribution 𝑃 (𝑦 | x 𝑖 ; Θ) calculated by the classifier 𝑓 : H → Y, the entire classification network can be trained by using loss functions such as cross-entropy loss, defined asEQUATIONwhere I 𝑦 * 𝑖 denotes the one-hot label vectors, and Θ is a set of model parameters. However, due to the closed-world property 𝐾 𝑖=1 𝑃 (𝑖 | x 𝑖 ; Θ) = 1, it can mistakenly classify novel class instances with high confidence. To identify instances from outside the predefined category set, a straightforward strategy is to set thresholds on the predicted prediction, the instance is conservatively labeled as "unknown" if the score is below the threshold 𝛿. Thus it can be inferred that the class predicted asEQUATIONHowever, simply applying manually defined thresholds to all known categories may not always be applicable or effective, especially when dealing with multi-view data with various representation spaces. Such an approach ignores the inherent complexity and diversity among different categories as well as among different views, which may lead to misclassification and poor recognition of novel categories. It's essential to consider a more tailored strategy that respects the complementarity and consistency nature of multi-view datasets to ensure robust classification and effective identification of new category instances beyond the known. 
the proposed framework: 
 The proposed model consists of four key components: 1) Soft-Mixup Augmentation Module introduces a soft labeling strategy for these synthetic interpolated data to measure their uncertainty. 2) Instance-level Sparse Inference Module employs a data-driven feedforward network to learn sparse approximated points in the multiple view embedding space; 3) View-level Confidence Network Module develops the view-level confidence map to approximate true class probability for the trustworthiness of predictions; 4) Centroid-Guided Adaptive Fusion Module adopts the interclass dispersion to metric the informativeness of each view for reliable fusion.3.0.1 Soft-Mixup Augmentation Module. Effective classifiers must distinguish known classes and adeptly manage the risk associated with "open space". This entails avoiding the overextension of class boundaries without cannibalizing the uncharted areas of the feature space that could belong to unrecognized categories. Drawing from the Mixup approach [25] , we apply linear interpolation on two samples drawn at random from the training data to produce additional virtual outliers. A random parameter 𝜆 ∈ [0, 1] is sampled as𝜆 ∼ Beta(𝛼, 𝛼). We select x (𝑣)𝑖 and x(𝑣)𝑗 in the 𝑣-th view to generate the augmented samples D 𝑚𝑖𝑥 = {{ X(𝑣) } 𝑉 𝑣=1 , ỹ} to enhance the generalization ability by familiarizing it with the open space near existing class clusters. For each view, we perform a randomized perturbation to selection x ′(𝑣) , which can be obtained as followsEQUATIONwhere 𝜆 determines the contribution of each original sample to features of the mixed samples. Since the samples produced by linear interpolation may lie in the neighborhood of another known class, simply assigning them to the unknown category may confuse the judgement of the predictor. Therefore, we employ a soft label I ỹ to teach the model to be uncertain for artificially created instances asEQUATIONSoft labels of mixed samples weight the distribution of class-specific metrics and all classes for an original sample, reflecting the probability that the sample belongs to each possible class. At the same time, by ensuring that unknown samples have a uniform probability distribution across known categories, the model avoids making overconfident predictions about never before learned categories. This approach mitigates potential classifier confusion by not strictly assigning these mix samples to an existing class but instead indicating their intermediate nature. Achieving a maximum entropy distribution across a set of categories indicates maximum uncertainty about which category a sample belongs to, implying that the model assigns probabilities across all categories as evenly as possible for unknown samples. Thus the learning objective for mixed samples is defined asEQUATIONUpon generating these mixed instances, the entire dataset represented as X (𝑣) = {X (𝑣) , X(𝑣) }, is fed into the proposed network. This process involves the integration of original and augmented data, enriching the learning context and bolstering its capability to discern and represent complex data patterns, ultimately leading EQUATION)where 𝜆 denotes a regularization parameter, and w = [𝑤 𝑖 ] 𝑉 𝑖=1 denotes a weight vector for balancing the contributions from distinct views. This formulation encapsulates the integration of data-fitting terms and various regularizations, aiming to implement efficient category separation in low-dimensional spaces.Specify, we first project diverse original feature spaces into unified dimensional space in preparation for subsequent fusion. To address the challenges inherent in high-dimensional problems, we turn our focus to the concept of model sparsity. Herein, we invoke the sparsity of the combined ℓ 1 -norm to suppress irrelevant features. Designating Φ as the view-specific projection matrix. Since the objective is a quadratic optimization with a non-differentiable regularizer problem. Formally, this corresponds to the case where the proximal operator defined for a convex regularizer, given byEQUATIONAt each iteration of the proximal algorithm, the current value of h is updated as the solution of the proximal problemEQUATIONwhere 𝐿 𝑓 is the Lipschitz constant of F . Taking inspiration from the Learned Iterative Shrinkage and Thresholding Algorithm network [8] , the optimization strategy can be effectively unrolled into a sequence of updates within a feedforward network. Here, the components that need to be precomputed are parameterized as a fully connected layer within the network, enabling it to learn adaptively from the data. To specifically address different views in the data, we extend our network layer update formulation as follows h(𝑣)EQUATIONwhereU (𝑣) 𝑡 = I -ΦΦ 𝑇 and V (𝑣) 𝑡= Φ 𝑇 serve to refine the viewspecific transformations at each iteration. And the update formula is defined as 𝜎 𝜏 (x) = F(x -𝜏) -F(-x -𝜏), where 𝜏 represents a learnable threshold parameter, and F(•) denotes an activation function such as ReLU, SELU, ELU, etc. The above formulation positions each network layer to correspond to an optimization algorithm iteration, enhancing the learning process of traditional optimization models through parameterized updates. Besides, we adopt a one-step inference, simplifying the process by retaining a single iteration of this optimization, thus promoting a balance between efficiency and performance.This module adopts a straightforward mathematical model inspired by sparse coding principles to construct the entire network structure. This choice is driven by two primary objectives: 1) To ensure that a sufficiently discriminative sparse representation is learned from both the original and mixed-sample features, reducing the interference of redundant information for improved adaptation to the open world. 2) To improve interpretability, with simpler models offering clearer insights into the dynamics between responses and covariates.3.0.3 View-level Confidence Network Module. Adopting the Maximum Softmax Probability MCP = 𝑃 (𝑦 = ŷ | x; Θ) as the confidence score usually leads to over-confidence, particularly in cases of incorrect predictions [3] . To counteract this issue, we consider the True Class Probability (TCP) as the target confidence value, defined as the predicted probability of the ground truth class TCP (x, 𝑦 * ) = 𝑃 (𝑦 = 𝑦 * | x; Θ). To obtain a confidence score estimate for each view, an auxiliary view-specific confidence network module 𝑐 (𝑣) parameterized by 𝜃 𝑐 is customized asEQUATIONwhere the sigmoid activate function is employed after the confidence network to ensure that CONF (𝑣) is normalized within the [0, 1]. CONF (𝑣) denotes a confidence map for each view, the goal of this module is to train it such that CONF (𝑣) closely approximates the TCP, utilizing a mean-square-error loss for this purposeEQUATIONTo further refine the capacity to leverage view-specific information effectively, a view-level gating strategy is employed asEQUATION)This approach ensures that the learned view-specific representations are informed by the reliability of predictions, thereby enhancing the overall predictive performance and trustworthiness.3.0.4 Centroid-Guided Adaptive Fusion Module. Recognizing the variability in quality across different views, where less informative views may dilute overall performance, we strategically assign weights to each view. Essentially, by emphasizing more informative views and diminishing the impact of less informative ones, we can achieve a more effective and reliable integration. Intuitively, the proximity of centroids within views is negatively correlated with the discriminative capability of each single view embedding. To quantify this, we first identify the centroids o (𝑣) 𝑖 of known category within the 𝑣-th view embedding space as belowEQUATIONwhere |𝐶 𝑖 | is the number of training instances in category 𝐶 𝑖 . Subsequently, we calculate the inter-category distance from all centroids of each view. To avoid making the largest distance between two classes overly influence the measure of interclass variation, we focus on the minimum distance between any two categories as a metric for view discriminativeness, denoted asEQUATIONwhere 𝐷𝑖𝑠𝑡 (•, •) is the distance function, which adopts the Euclidean distance in this work. This strategy provides a balanced assessment of the contribution of the different views. Building on this Genrate the augmented samples { X(𝑣) ∈ R 𝑚×𝐷 (𝑣), ỹ} for joint training as Eq. ( 4); 5:Calculate view-specify representation H (𝑣) as Eq. ( 9); 6:Compute confidence map CONF (𝑣) though view-specific confidence network module as Eq. ( 10); Update H (𝑣) with view-level gating CONF (𝑣) as Eq. ( 12); 8:Calculate the weight of multiple views 𝑤 (𝑣) as Eq. ( 15); Fusing multiple view representations to obtain viewconsistent representation H as Eq. ( 16); Compute the loss of joint optimization as Eq. ( 17); 1 . By normalizing the reciprocal of these distances, views with more distinctive category separations are afforded greater weights. Accordingly, the weight assignments w can be determined asEQUATION)Ultimately, these weights are employed to construct the multi-view consistent representation H through a weighted summation of individual view embeddingsEQUATIONThrough this method to achieve balanced and insightful fusion of multiple views, this embedding is expected to be more discriminative than any single-view setting.Additional classifier, denoted as 𝑓 : H → Y, is tailored to process view-consistent representation H. The output from the final fullyconnected layer is subsequently processed through the SoftMax function, generating a probability distribution over the labels of all 𝐾 recognized classes. The overall learning objective of the model is encapsulated by the following composite loss functionEQUATION)L 𝑐𝑒 plays a fundamental role in guiding the model toward the accurate classification of known categories on the training data. L 𝑐𝑜𝑛𝑓 facilitates the alignment of the output of the confidence network with the true confidence level as indicated by TCP, thereby enhancing the reliability, particularly in its confidence estimations. L 𝑠𝑜 𝑓 𝑡 is crafted to minimize prediction error on these mixed instances, enhancing its ability to generalize from known to unknown categories. This objective not only enhances confidence accuracy but also improves adaptability to unknown categories, ensuring robust and versatile performance in various real-world applications. The entire procedure is outlined in Algorithm 1.The core computations for computing multi-view consistent representation are 1) creation of 𝑀 mixed samples where linear interpolation is O (𝑀𝐷 (𝑣) ); 2) one-step sparse inference, for which the complexity is O ((𝑁 + 𝑀)𝐻2 + (𝑁 + 𝑀)(𝐷 (𝑣) ) 2 ), accounting for operations on both the original 𝑁 samples and the 𝑀 mixed samples.; 3) the gating strategy is O ((𝑁 + 𝑀)𝐻 ), and 4) Computing the centroids for each category has a complexity of O ( 𝐾 𝑖=1 |𝐶 𝑖 |𝐷 (𝑣) ) = O (𝑁 𝐷 (𝑣) ) and calculating inter-category distances between 𝐾 (𝐾 + 1)/2 pairs of centroids is O (𝐾 2 𝐷 (𝑣) /2). Given that 𝑁 is typically larger than 𝑀, the overall time complexity becomes O (𝑉 𝑁 𝐻 2 + 𝑉 𝑣=1 (𝑁 (𝐷 (𝑣) ) 2 ) + 𝐾 2 𝐷 (𝑣) )) under multi-view setting for each training epoch. 
experimental results and study 4.1 benchmark datasets: 
 Our experiments are conducted on six well-known multi-view datasets. The statistics of these datasets are summarized in Table 1 , with detailed descriptions provided below.1) ESP-Game1 is a social image collection searched from an image annotation website where two players without any communication try to predict the same tags for a test image. 2) Flower17 2 consists of different flower categories that are common in the UK. The images are available in a variety of views, scales and photometric variations. 3) MNIST3 is a well-known dataset of handwritten digits with IsoProjection, LDA, and NPE features. 4) ORL4 is a face image dataset, taken at varying lighting, different times, facial expressions, and facial details. 5) Reuters5 consists of multilingual document corpora in five languages: English, French, German, Spanish, and Italian. 6) Youtube6 is a video game dataset including both visual (Cuboids Histogram, Histogram of Motion Estimate, and HOG features) and audio features (MFCC, Volume Streams, and Spectrogram Streams). We compare the proposed method with several state-of-the-art methods for multi-view representation learning, including 1) IMvGCN [21] combines the reconstruction error and Laplacian embedding, accompanied by a differentiable orthogonal normalization constrain to improve generalization capability. 2) LGCN-FF [1] integrates sparse autoencoders with a learnable GCN, enabling the simultaneous extraction of feature representations and node relationships within graphs. 3) MMDynamics [9] models both the feature-level and modality-level informativeness for trustworthy fusion. 4) MvNNcor [23] models view-specific information and cross-correlations information through an interactive network to jointly make decisions and infer categories. 5) PDMF [22] earns relations and the auxiliary representation through pre-training to tune the mappings from the original data to the comprehensive representation. 6) TMC [10] conducts decision fusion according to uncertainty estimation of multiple views.To handle known and unknown samples separately, we introduce the Open-Set Classification Rate (OSCR) [4] . By setting a probability threshold 𝜃 , it can balance sensitivity to known classes with the ability to reject unknowns. For samples from known categories, we calculate the Correct Classification Rate (CCR) as the fraction of the samples where the correct class 𝑦 * has maximum probability and has a probability greater than 𝜃 . Concurrently, False Positive Rate (FPR) is determined by the fraction of samples from the unknown category that are classified as any known class 𝑦 ∈ Y with a probability greater than 𝜃 . Different applications may tolerate different levels of FPR for the benefit of higher CCR, making these specific metrics highly relevant for tuning the models according to specific needs. For security-critical applications such as biometrics or fraud detection, a low FPR is prioritized over CCR to block unauthorized access. Conversely, content recommendation or advertising systems may allow a higher FPR to enhance inclusivity and user experience by ensuring relevant content is not overlooked. In order to measure the quality of representation, we evaluate model performance with CCR at FPR of 0.5%/1.0%/5.0%/10.0%. Regarding the choice of thresholds, which is randomized, we follow the experimental setup in [4] by taking the maximum probability set of predicted unknown samples and traversing it to compute the CCR and FPR.In our experimental setting, we use the concept of openness [15] ,EQUATIONwhere 𝐶 𝑡𝑟𝑎𝑖𝑛 is the number of known classes during training, and 𝐶 𝑡𝑒𝑠𝑡 is the total number of known and unknown classes during testing. The proposed AAML has been implemented using the Py-Torch framework on one NVIDIA Geforce RTX 4080 with GPU of 16GB memory. In practical application, rather than pre-generating all mixing samples, we adopt linearly transforming each mini-batch during the training process. Our training protocol involves 100 epochs across all benchmarks, maintaining a batch size of 100. In In all the experiments below, all our open experiments have compared the proposed AAML with other classification methods under the condition of openness=0.1.1) The OSCR curves of different methods on six datasets are depicted in Fig. 2 . The proposed method (red solid line) overwhelmingly outperforms the other methods (colored dashed lines) in all cases, particularly notable in the Flower17 and Youtube datasets, which demonstrates its ability to maintain a high CCR while controlling the FPR. Specifically, it excels in minimizing misclassifications of unknown categories while accurately classifying known classes. This capability ensures that our model is adaptable to different applications with varying FPR requirements, making it a versatile and effective solution.2) In order to conduct a numerical analysis, we computed the values of CCR at several representative FPR values, organized in Tables Table 4: Classification accuracy of all compared methods on multi-view datasets under openness=0 setting. 3) In addition to the above results based on the open setting, we report the accuracy of compared methods on each dataset with the openness equals 0in Table 4 , where the problem is completely closed. The outcomes demonstrate that the proposed model maintains competitive classification performance on known classes, affirming its capability to adeptly navigate the heterogeneity and redundancy inherent in multi-view data. 4) Fig. 3 presents scatter diagrams of AAML with different openness on MNIST. while colored dots signify samples from known classes, while red crosses indicate samples from unknown classes. The results intuitively highlight that the classes, as defined by the known labels, exhibit greater compactness and distinct separation. It is evident that samples belonging to the unknown class predominantly cluster around the centroid, albeit with two classes seemingly merging with it. In contrast, the other classes are situated farther away from the unknown class, emphasizing their distinctness. configuration involves directly employing the average fusion strategy, bypassing the centroid-guided adaptive fusion module. We have the following observations from Fig. 4 : AAML (w/o S) exerts the most significant impact on the ESP-Game and ORL datasets. AAML (w/o C)demonstrates a modest decrease across all datasets, although its effect is less pronounced compared to module S. On the other hand, AAML (w/o F ) exhibits the smallest impact, resulting in only a slight decrease in performance across all datasets. Despite the varying degrees of impact observed across these modules, our comprehensive analysis underscores the importance of AAML, as it consistently outperforms the ablated versions across all datasets. This suggests that the synergistic integration of all modules leads to the best overall performance.4.6.2 Parameter Analysis. To investigate the influence of the tradeoff parameter 𝛼 and 𝛽 in Eq. ( 17), we conduct a sensitivity test by varying from 0.001 to 100. Fig. 5 shows the influence of different parameter values with respect to CCR at FPR at 10% on the ESP-Game and MNIST datasets. Moreover, our method performs stably when the values of 𝛼 and 𝛽 are within a certain region. 
conclusion: 
 Under open multi-view learning contexts, characterized by ambiguity, we confront two principal challenges: Mitigating the adverse effects of variability and recognizing unknown classes with multiview information. In this paper, the inherent ambiguity of these synthetic points is managed by creating samples located at taxon boundaries, using mixed samples from different taxa, and applying soft labels to manage the inherent ambiguity of these synthetic points. In this way, the generalization ability of the model is enhanced by adapting the model to the data distribution of potentially unknown categories in advance during the training phase. In addition, we design a one-step sparse inference model that proficiently eliminates superfluous information across multiple views. This is complemented by confidence score estimation, which gates viewspecific representations to leverage prediction scores and enhance reliability. Additionally, the inter-class dispersion is adopted to ascertain the informativeness of the views, thereby amalgamating them into a cohesive final representation. The proposed AAML not only accurately classifies instances based on comprehensive multiview data, but also extends its classification capabilities beyond the training data. 
 
 Appendix: https://www.cs.cmu.edu/~biglou/resources/http://www.robots.ox.ac.uk/vgg/data/flowers/http://yann.lecun.com/exdb/mnist/http://cam-orl.co.uk/facedatabase.htmlhttps://archive.ics.uci.edu/dataset/259/reuters+rcv1+rcv2+multilingual+multiview+ text+categorization+test+collectionhttps://archive.ics.uci.edu/dataset/269/youtube+multiview+video+games+dataset 


Decision: All reviewers lean towards acceptance of this paper. The AC concurs with the reviewers and recommends this paper for publication. The authors should address all the issues raised in the reviews in the final version of their paper.

Reviews: review: paper_summary: paper_summary: This paper focuses on the practical problem of considerable ambiguity among open multi-view learning and propose a Ambiguity-Aware Multi-view Learning Framework (AAML), which consists of four key components that each geared towards achieving specific objectives to jointly address the unknown classes problem. Experimental results on six well-known multi-view datasets demonstrate that the effectiveness of AAML.
strength_weakness: strength_weakness: 1. The studied problem is interesting.
2. This paper is well-written, well-organized, and easy to follow.
limitations: limitations: 1. In the INTRODUCTION, the authors propose two questions. However, based on the narrative flow of the paper, the first question appears to be somewhat overlooked. It is suggested that the authors revisit the introduction and ensure the first question is adequately incorporated into the logical flow of the paper.
2. In the RELATED WORK section, some related works are missing: a) CL-MVSNet: Unsupervised Multi-view Stereo with
Dual-level Contrastive Learning, ICCV 2023; b) Self-Supervised Information Bottleneck for Deep Multi-View Subspace Clustering, TIP, 2023
3. The paper could be strengthened by providing a more detailed analysis of the effectiveness and limitations of the proposed approach. This factor could be significant for guiding future research and practical applications of the methodology.
paper_summary: paper_summary: The paper introduces an innovative framework to address the challenge of ambiguity in open multi-view learning scenarios. The authors propose an Ambiguity-Aware Multi-view Learning Framework (AAML) that integrates four key modules: Soft-Mixup Augmentation, Instance-level Sparse Inference, View-level Confidence Network, and Centroid-Guided Adaptive Fusion. The framework aims to enhance the generalizability and reliability of predictors by adapting to the distribution of unknown samples and reconciling disparate views to distill essential patterns.
strength_weakness: strength_weakness: - The paper introduces an Ambiguity-Aware Multi-view Learning Framework (AAML) that addresses the challenge of ambiguity in open multi-view learning scenarios. This represents a novel approach to handling variability and unpredictability in real-world data.
- The framework is designed to adapt to the distribution of potentially unknown samples during training, which enhances its ability to generalize beyond the known categories and makes it robust for open-world recognition tasks.
- AAML specifically addresses the issue of ambiguity, which is often neglected in previous works but is crucial for real-world applications where models must operate under conditions that may differ from those seen during training.
limitations: limitations: - How does the concept of ambiguity in multi-view learning differ from uncertainty in single-view scenarios, and why is it crucial to address this?
- Can the authors elaborate on why mixed samples and soft labels are essential for adapting to the distribution of unknown samples? Meanwhile, I think that it may introduce more noise and uncertainty to the learning process.
- Are there existing methods or frameworks that the authors found inadequate in handling ambiguity in multi-view learning, and how does their work differ?
- Can the authors elaborate on the sparse inference process and how it contributes to filtering out redundant features? How is the sparsity enforced, and what is the role of the combined ℓ1-norm in this context?
- The paper introduces the concept of ambiguity in multi-view learning. I am confused that how is ambiguity specifically quantified and handled within the framework? 
- What criteria were used to select the benchmark datasets (ESP-Game, Flower17, MNIST, Reuters, ORL, Youtube), and how do they represent the challenges of open multi-view learning? 

- The paper mentions managing uncertainty with soft labels. How is uncertainty quantified in the experiments, and what is the impact on the decision-making process?
paper_summary: paper_summary: This paper addresses challenges in open multi-view learning, such as handling variability and recognizing unknown classes. By creating synthetic samples at taxon boundaries and using soft labels, the model's generalization to unknown categories is enhanced. The paper introduces a one-step sparse inference model that removes unnecessary multi-view information, complemented by a confidence scoring system to improve prediction reliability. Additionally, the method evaluates and integrates the informativeness of different views into a cohesive representation. The proposed framework not only classifies multi-view data accurately but is also adapted to recognize categories beyond the training data.
strength_weakness: strength_weakness: 1. This paper is well written and easy to understand.
2. Good experimental results on four scenarios.
limitations: limitations: It is suggested that future revisions include a deeper exploration of the motivations behind specific design decisions, enhancing the reader's understanding of the reasons for these choices beyond just a description of the implemented actions.
paper_summary: paper_summary: The paper introduces the Ambiguity-Aware Multi-view Learning Framework (AAML), designed to handle the inherent ambiguity in open multi-view learning environments. The framework integrates four synergistic modules to enhance generalizability and reliability by creating mixed samples with soft labels, implementing instance-level sparse inference, generating view-level confidence mappings, and employing centroid-guided adaptive fusion for consistent representation. This approach effectively prepares the model for unknown categories and improves classification accuracy and robustness. Extensive experiments validate the superior performance and stability of AAML compared to other state-of-the-art methods.
strength_weakness: strength_weakness: - The AAML integrates multiple synergistic modules to handle complexity in a systematic way, which is a strength in terms of providing a holistic solution. The centroid-guided adaptive fusion dynamically adjusts the contributions of different views, ensuring a cohesive and informative final representation.
- Extensive comparative experiments demonstrate AAML's superior performance and stability in open-world recognition scenarios.
limitations: limitations: 1. The motivation behind the paper is not clearly articulated. The concept of ambiguity is not well-defined within the context of the paper. It is unclear whether ambiguity refers solely to unknown classes/outliers, or if it encompasses other aspects. This lack of clarity could make the title seem misleading or out of scope relative to the actual content and contributions of the work.

2. The paper claims that linear interpolation of two samples can produce outliers. However, linear interpolation should typically result in a combination of two existing images from the original distribution, not outliers. If this approach were to generate outliers, it would imply that existing mixup-based methods would fail, which contradicts current empirical evidence supporting their effectiveness.

3. How to combine Soft-Mixup Augmentation with the ground truth class TCP? How to obtain the predicted probability of the ground truth class TCP?

4. The paper lacks comprehensive comparisons with the methods like "Robust Multi-view Clustering with Incomplete Information" (TPAMI 2023) and "Dual Contrastive Prediction for Incomplete Multi-View Representation Learning" (TPAMI 2023).

5. The writing is not well-organized, and the annotations used in the equations are confusing. For example, the difference between $\mathbb{I} \tilde{y}$ and $\mathbb{I} \tilde{y}_i$ in Equations (4) and (5) is not explained. Additionally, the same notation $\lambda$ is used in both Equations (3) and (6) without clarification, potentially leading to misunderstandings. The meaning of $\Phi$ in Equation (9) is also not defined.