Learning Unknowns from Unknowns: Diversified Negative Prototypes Generator for Few-shot Open-Set Recognition

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-31 00:01:12

Abstract: Few-shot open-set recognition (FSOR) is a challenging task that requires a model to recognize known classes and identify unknown classes with limited labeled data. Existing approaches, particularly Negative-Prototype-Based methods, generate negative prototypes based solely on known class data. However, as the unknown space is infinite while the known-space is limited, these methods suffer from limited representation capability. To address this limitation, we propose a novel approach, termed Diversified Negative Prototypes Generator (DNPG), which adopts the principle of "learning unknowns from unknowns." Our method leverages the unknown space information learned from base classes to generate more representative negative prototypes for novel classes. During the pre-training phase, we learn the unknown space representation of the base classes. This representation, along with inter-class relationships, is then utilized in the meta-learning process to construct negative prototypes for novel classes. To prevent prototype collapse and ensure adaptability to varying data compositions, we introduce the Swap Alignment (SA) module. Our DNPG model, by learning from the unknown space, generates negative prototypes that cover a broader unknown space, thereby achieving state-of-the-art performance on three standard FSOR datasets. We provide the source code in the supplementary materials for reproducibility.

Main paper: 
 related work: 
 Few-Shot Learning. Few-shot learning in visual classification addresses the challenge of classifying with limited support samples. Methods can be broadly categorized into transfer learning, which leverages pre-trained models from related tasks [33, 37] , and metalearning, which includes model-based [4, 23] , metric-based [12, 30] , and optimization-based approaches [1, 16] . Model-based methods adapt quickly to new tasks with minimal data, metric-based methods optimize the distance distribution between samples, and optimization-based methods focus on efficient model training with limited examples.Open-Set Recognition. Current open-set recognition methods can be categorized based on their approach to discriminating unknown class samples, including similarity threshold-based methods [2, 8, 29] , and density threshold-based methods [20, 39] . Additionally, a new trend has emerged, where Gaussian Mixture Models (GMMs) are applied to model closed-set distributions [7, 21] , enabling the models to directly reject unknown class samples. However, no previous work has employed GMMs to address the FSOR task. This is mainly attributed to the difficulty of learning the joint distribution of classes and pixel features when only a limited number of labeled samples from the novel classes are available.Few-Shot Open-set Recognition. FSOR combine the challenges of FSL and OSR, requiring the completion of OSR tasks with limited support samples. PEELER [22] proposes to train and test FSOR tasks in a meta-learning pattern. RFDNet [10] suggests meta-learning a feature displacement relative to a pre-trained reference feature embedding. SnaTCHer [17] identifies unknown samples based on the transformation consistency, which measures the difference between transformed prototypes and a modified prototype set. The GEL [35] model utilizes an energy function as the discriminative criterion and introduces a pixel-level prototype network. ReFOCS [24] adopts a generative approach to dynamically adjust the similarity threshold by reconstructing samples. OPP [32] proposes constructing an overall positive prototype and then filters out unknown class samples using a threshold method. MSCL [40] combines the strengths of supervised contrastive learning and meta-learning to effectively increase inter-class distinctions and reinforce intra-class compactness. MRM [5] proposes enlarging the margin between different classes by extracting the multi-relationship of paired samples to dynamically refine the decision boundary for known classes and implicitly delineate the distribution of unknowns.Negative-Prototype-Based FSOR Methods. These FSOR methods learns one or multiple prototypes for unknown class samples,thereby transforming the open-set recognition task into an N+1 classification task. Existing Negative-Prototype-Based FSOR methods: ProCAM [31] learns NPs by extracting the background of support set images, ATTG [15] uses attention mechanisms to generate NPs that are distant from known class samples. ASOP [18] proposes to generate a negative prototype closest to open-set samples and second-closest to closed-set samples. TNPNet [38] adopts a transductive learning approach, leveraging both local and global contexts to enhance prototypes and negative prototypes. However, these methods utilize known class samples from the support set to learn NPs, which inevitably leads to the NPs inheriting the characteristics of the known class samples. As a consequence, the learned NPs are limited in their ability to model the unknown space, as illustrated in Figure 2 (a) . Multiple Prototypes Model. This approach involves generating multiple prototypes to represent a batch of data. Methods like [6, 15] generate multiple negative prototypes, but often face challenges such as prototype collapse and ineffective utilization of prototypes. Some studies [25, 36] attempt to address these issues with an equipartition constraint, but these methods may not perform well in small-sample open-set recognition tasks and could potentially decrease model performance. A proficient FSOR model should: (1) accurately classify images from the query set 𝑄 𝑘 into the corresponding known classes using the knowledge encapsulated in the support set S, and (2) effectively recognize images from classes not represented in the current support set (i.e., samples from 𝑄 𝑢 ) as belonging to unknown classes. 
negative-prototype-based fsor: 
 Negative-prototype-based FSOR approaches transform the open-set recognition problem into an (𝑁 + 1)-way closed-set classification task by introducing Negative Prototypes (NPs). In the absence of unknown-class samples, a prototype-based few-shot classifier can be formulated asEQUATIONwhere F 𝜃 is the feature extractor, 𝑓 𝑠 is the similarity fynction, C 𝑘 denotes the set of novel known classes, and p 𝑐 represents the prototype for class 𝑐, computed as the average feature vector of samples belonging to class 𝑐. To accommodate unknown-class samples, Negative-Prototype-Based FSOR models extend the prototypebased classification by learning an additional NP for the unknown classes. The classification then involves (|C 𝑘 | + 1) prototypes:EQUATIONwhere a sample is classified as belonging to an unknown class if it is most similar to the NP. However, as illustrated in Figure 2 (a), existing approaches often derive NPs based on support samples, inadvertently incorporating information from known classes. This can lead to NPs that closely resemble known classes or highly similar classes (e.g., Dalmatians and Alaskans), which complicates the modeling of unknown classes within the diverse space of unknowns. 
the proposed method: 
 In this section, we introduce our approach to tackle the FSOR problem by generating diverse NPs. Our model architecture is depicted in Figure 3 . We first describe our baseline model, followed by our strategy for learning open-set weights for base classes. Finally, we present our Diversified Multiple Negative Prototype Generator (DMNPG) module and the Swap Alignment module, which together facilitate the generation of task-level diversified NPs for different tasks.Our baseline model is built upon the framework proposed by Huang et al. [15] and utilizes a metric-based meta-learning architecture, similar to previous FSOR methods [12, 15, 30] . As shown in Figure 1 , the training process is divided into two stages: the base-class stage, which uses a large-scale dataset of base classes, and the novel-class stage, which evaluates the model on a dataset of novel classes with all model parameters fixed. Pre-training. During pre-training, we employ a ResNet-12 network to perform a large-scale classification task using labeled samples from the base classes. The resulting feature extractor F 𝜃 and the classifier head for base classes are obtained, with the classifier head weights denoted as 𝑃 * . Meta-learning. In the meta-learning stage, the base class dataset is redivided into different tasks. Sampling 2N classes in each task, with N known classes (S and Q 𝑘 ) and N designated as pseudo-unknown classes (Q 𝑢 ). The pre-trained Res-12 network serves as the feature extractor F 𝜃 to extract low-dimensional feature representations 𝑣 𝑖 for samples in each task. For each task, the feature representations in the support set are averaged by class to obtain the raw class prototype 𝑝 𝑟 𝑐 :EQUATIONwhere 𝑥 S 𝑐,𝑖 represents the 𝑖-th sample of class 𝑐 from the support set, and 𝑝 𝑟 𝑐 ∈ 𝑃 𝑟 is the raw class prototype of class 𝑐, with 𝑐 ∈ C 𝑆 . A Multi-Layer Perceptron (MLP) is then applied to the averaged class prototype 𝑝 - 𝑎𝑣𝑔 to generate the NP:EQUATIONwhere 𝑝 - 𝑎𝑣𝑔 is the average of the raw class prototypes, considering all novel classes in the current task to generate a task-specific NP.After obtaining both the raw class prototypes 𝑃 𝑟 and the NP 𝑝 -, the closed-set classification and open-set recognition tasks are EQUATIONwhere 𝑓 𝑠 is a function to measure the similarity between two inputs, e.g., cosine similarity. The cross-entropy loss is used to train the (|C 𝑘 |+1)-way classification baseline model asL 𝐶𝐸 (Q 𝑘 ∪ Q 𝑢 ).Testing. During the testing phase, the model parameters are fixed. For a test task T = {𝑆, 𝑄 𝑘 , 𝑄 𝑢 }, prototypes of novel classes and the NP are generated using the support set samples S, as per the equations above. The cosine similarity between the query samples {𝑄 𝑘 , 𝑄 𝑢 } and the prototypes, including the NP, is calculated. A query sample is classified into the corresponding novel class if its cosine similarity with a novel class prototype is the highest. However, if its cosine similarity with the NP is the highest, it is considered to be from an unknown class.Our method aims to generate negative prototypes directly from the unknown space by utilizing the base-class inverse representation, referred to as open weights, and leveraging the similarity relationships between classes.Learning the transferable open weights. In the pre-training phase, we adopt the concept of Reciprocal Prototype Learning (RPL) [6] to learn the open weight 𝑂 𝑗 for each base class 𝑗. The objective is to position 𝑂 𝑗 far from the samples of class 𝑗 while maintaining proximity to samples from other base classes. This is achieved through a reverse classification task where the probability of a sample being classified as class 𝑗 is inversely proportional to its distance from 𝑂 𝑗 :EQUATION)After obtaining the classification probability, the cross entropy loss function is used to train the model:EQUATIONwhere Raw Class Prototypes Calibration Inspired by [11] , we refine the raw class prototypes 𝑃 𝑟 for novel classes during the metalearning phase using the base class weights 𝑃 * (refer to Figure 4 , left). Specifically, we adjust 𝑃 𝑟 to incorporate characteristics akin to 𝑃 * , resulting in the final class prototypes 𝑃:EQUATIONEQUATIONwhere 𝑊 𝑞 ,𝑊 𝑘 ,𝑊 𝑣 ∈ R 𝑑 ×𝑑 are trainable parameters, and 𝜎 denotes the softmax operation. The resulting weight matrixA (𝑃 𝑟 ,𝑃 * ) ∈ R | C 𝑆 | × | C 𝐵 |captures the learned relationships between novel and base classes for task 𝑇 .The open weights, representing the unknown space of base classes, reveal that samples from similar classes exhibit analogous characteristics in this space (as illustrated in Figure 5 ). Exploiting this observation, we generate Negative Prototypes (NPs) for novel classes using the open weights of base classes (see Figure 4 , right). Specifically, leveraging the class similarity matrix A (𝑃 𝑟 ,𝑃 * ) from Eq. 8, we construct the NPs by allowing novel classes to mimic the unknown space characteristics of base classes, while preserving the class similarity relationships. The NPs for each novel class are generated as follows:EQUATIONwhere 𝑊 𝑣 is a trainable parameter, and 𝑂 and A (𝑃 𝑟 ,𝑃 * ) represent the open weights of base classes. Each NP, 𝑃 - 𝑗 ∈ 𝑃 -, signifies the negation of novel class 𝑗. To optimize 𝑃 - 𝑗 , we employ a binary crossentropy loss that minimizes its similarity with queries of class 𝑗 and maximizes its similarity with queries from other classes:EQUATIONThen, we utilize the NP 𝑃 - 𝑗 ∈ 𝑃 -of each novel class to replace the input 𝑝 𝑟 𝑐 in Eq. 4, resulting in the NP 𝑝 -for the entire task. Considering the diverse appearances of unknown samples (e.g., both dogs and tigers can be considered unknown for the 'cat' class), we propose employing multiple NPs for each task. Specifically, we apply 𝑁 MLPs to Eq. 4, yielding 𝑁 distinct NPs:EQUATION)Although we have generated multiple NPs for each task, not all NPs are effective, as detailed in Section 5.4. To address this, we propose the Swap Alignment Module (SA) to facilitate the generation of diverse and suitable NPs for tasks involving different classes. Inspired by ATTG [15] , we employ a conjugate method to sample a pair of tasksT 1 = 𝑆 1 , 𝑄 1 𝑘 , 𝑄 1 𝑢 | 𝐶 1 𝑘 and T 2 = 𝑆 2 , 𝑄 2 𝑘 , 𝑄 2 𝑢 | 𝐶 2𝑘 , with the property that 𝑄 1 𝑘 = 𝑄 2 𝑢 and 𝑄 1 𝑢 = 𝑄 2 𝑘 . This implies that the known few-shot classes 𝐶 1 𝑘 in T 1 and 𝐶 2 𝑘 in T 2 serve as negative sources for each other. We refer to T 1 and T 2 as each other's swap tasks.For these swap tasks, our model generates prototypes and NPs, denoted as [𝑃 1 , 𝑃 - 1 ] and [𝑃 2 , 𝑃 - 2 ], respectively. We then apply the swap alignment operations :EQUATIONEQUATIONThe GCN is a lightweight GCN network used for information propagation between prototypes and NPs, 𝑃 𝑗 2,𝑎 is the prototype of the 𝑗-th novel class in task T 2 , and 𝑃 𝑖,-1,𝑎 is the 𝑖-th generated negative prototype in task T 1 . For each pair of swap tasks, we minimize the distance between each negative prototype in T 1 and the most similar novel prototype in T 2 , and vice versa. This alignment ensures that the unknown classes in T 1 (represented by the negative prototypes) correspond to the novel classes in T 2 (represented by the prototypes). The alignment loss L align 𝑛 aims to bring the NPs of task T 1 closer to the prototypes of task T 2 , as they represent the same set of samples (𝑄 1 𝑢 = 𝑄 2 𝑘 ). By reducing the distance between NPs and prototypes, the NPs become more discriminative and better approximate the distribution of real samples.For task T 1 in the swap task pair (T 1 , T 2 ), the final training loss function is:EQUATIONwhere 𝛼 and 𝛽 are hyperparameters. Similarly, we calculate L T 2 for task T 2 . The total swap training loss is L = L T 1 + L T 2 . 
experiments 5.1 datasets and implementation details: 
 Datasets. We conduct FSOR experiments are conducted on three widely-used few-shot learning datasets: MiniImageNet [34] , Tiered-ImageNet [26] , and CIFAR-FS [3] . MiniImageNet is a subset of ILSVRC-12 [28] , consisting of 100 categories with 600 images each. The dataset is divided into meta-training, meta-validation, and metatesting sets with 64, 16, and 20 categories, respectively. TieredIma-geNet, also a subset of ILSVRC-12, contains 608 categories divided into 351, 97, and 160 categories for the respective sets. CIFAR-FS is derived from CIFAR100 [19] , comprising 60,000 images across 100 categories, with the same division scheme as MiniImageNet. Implementation Details. Following the settings of GEL [35] and ATTG [15] , we employ ResNet-12 as the feature extractor. During the pre-training phase, we follow the protocol of Tian et al. [33] and Huang et al. [15] , pre-training ResNet-12 and a linear classifier with a combination of cross-entropy loss and self-supervised rotation loss on the base set for 90 epochs. We use SGD as the optimizer, with an initial learning rate of 0.05, decayed by a factor of 10 at epoch 60. In the meta-training phase, the learning rate is set to 0.0001 for ResNet-12 and 0.05 for all other layers in the negative prototype generator. For FSOR testing and evaluation, we follow the task sampling strategy of Liu et al. [22] , setting 𝑁 = 5 and 𝐾 = 1, 5. Each task includes 15 positive queries from each few-shot class and 5 negative classes, each with 15 negative queries. We employ cosine similarity as the similarity function, utilizing two different temperature coefficients for comparing samples with prototypes (𝑠𝑖𝑚1) and negative prototypes (𝑠𝑖𝑚2). Additionally, a trainable bias is added to 𝑠𝑖𝑚2, allowing for an adjustable similarity offset for all samples to the negative prototype. Classification into the sixth class (representing the unknown class) is based on Equation 3, with the sample's probability of belonging to this class determined accordingly. The hyperparameters 𝛼 and 𝛽 are both set to 1.Metrics. Consistent with prior work [10, 15, 31, 35] , we evaluate closed-set classification performance using the accuracy metric (ACC) and open-set recognition performance using the Area Under the ROC Curve (AUROC). Higher values of ACC and AUROC indicate superior performance.We evaluate our model against state-of-the-art (SOTA) few-shot open-set recognition approaches, which can be categorized into two groups: Negative-Prototype-Based methods, including ATT, ATT-G [15] , and ASOP-L [18] , and threshold-based models, such as PEELER [22] , SnaTCHer [17] , RFDNet [10] , and GEL [35] . Notably, PEELER and RFDNet originally employ Res-18 as the feature extractor, but Wang et al. [35] implemented Res-12 versions, denoted as PEELER* and RFDNet*, whose results we directly quote for comparison. For other models, we use the results reported in their respective papers. Tables 1 and 2 present our model's performance compared to other approaches on three datasets. Our method surpasses all competing methods in both 1-shot and 5-shot settings across all datasets. Additionally, on the COIL-DEL graph dataset [27] , our model demonstrates strong FSOR performance, as shown in Table 3 . It is important to note that the innovation of our model lies in its ability to learn unknowns from unknowns. This is highlighted by the fact that our model consistently outperforms the ATTG model, which also utilizes information from both novel and base classes during testing, on all datasets. This indicates that our approach of learning unknowns from unknowns is indeed effective for the FSOR task. Given that the ATTG model outperforms the ATT model, we select ATTG as the representative of existing Negative-Prototype-Based FSOR models in subsequent experiments.To further demonstrate the superiority of our DNPG model, we provide visualizations of prototypes and NPs generated by both the ATTG and DNPG models in Figure 6 . For both models, we set 𝑁 = 8, meaning that 8 MLPs are used to generate 8 NPs for each task. It can be observed that the NPs generated by the ATTG model, which relies on support samples as inputs, tend to erroneously converge towards the prototypes of the corresponding classes, thereby losing their ability to effectively model unknown samples. Conversely, our DNPG model does not suffer from this issue, maintaining the distinctiveness of NPs from the prototypes.To assess the contribution of each component in the DNPG model, we conducted an ablation study with 𝑁 = 5, meaning that 5 MLPs are used to generate 5 NPs for each task. In this context, MNG refers to the proposed NPs generator. Although the RPC module is not a novel contribution of our work, it is an integral part of our model; thus, for fairness, we also present results for the baseline combined with RPC. SA denotes the Swap Alignment Module. Since the SA module is designed based on Conjugate Training (CT) [15] , we include an ablation model, baseline + RPC + MNG + CT, which excludes the SA module and solely employs conjugate training.Table 4 shows the performance of each ablation model on the MiniImageNet dataset for 5-way 1-shot and 5-way 5-shot FSOR tasks. The results demonstrate that the DNPG model significantly outperforms the baseline in both ACC and AUROC metrics. Additionally, each individual module contributes positively to the overall performance of the model.By comparing the baseline + RPC + MNG + CT model with the baseline + RPC + MNG + SA model, it is evident that conjugate training does not markedly enhance our model. This is because conjugate training is aimed at mitigating the overfitting of generated NPs to the pseudo-unknown class samples. However, our model does not rely on pseudo-unknown class samples for generating NPs, thereby circumventing the overfitting issue. In contrast, the SA module significantly boosts the performance of our model, underscoring its importance in the DNPG framework.Table 4 : Ablation study of our model. We report the 5-way 1-shot and 5-way 5-shot results on MiniImageNet. Our innovations are bolded. Better Negative Prototypes, Wider Unknown Space. To further validate that our model's NPs contain minimal known class information and can model a broader unknown space, we conduct experiments using two distinct datasets to generate NPs and analyze their differences.We train three FSOR models, ATTG [15] , GEL [35] , and our DNPG, on the training set of MiniImageNet. After fixing all parameters, we evaluate the models on both the test set of MiniImageNet and the test set of CIFAR-FS. We apply Principal Component Analysis (PCA) to reduce the dimensionality of the generated NPs in ATTG and DNPG and visualize them. Since GEL is not a Negative-Prototype-Based model, it does not generate NPs.Figures 7a and 7b show the distributions of prototypes and NPs generated by the ATTG and DNPG models for both MiniImageNet and CIFAR-FS datasets. The NPs generated by the ATTG model display noticeable differences between the two datasets, indicating the incorporation of dataset-specific information. In contrast, our DNPG model produces universal NPs that effectively model a wide range of unknown spaces, making it difficult to distinguish the learned NPs between the two datasets.Lastly, we compare the ability of different models to fit unknown class data using the learned NPs. Specifically, we calculate the maximum similarity between each unknown class sample and the NPs during testing, as depicted in Figure 8 . It is evident that the NPs learned by our model exhibit superior ability to fit various How does the SA Module Work? To enhance the model's adaptability to tasks composed of diverse data, we generate multiple NPs for each task. Existing methods, such as ATTG [15] , use the distance between a sample and its most similar NP as the classification score for the unknown class during both training and testing. However, simply employing multiple NPs does not ensure sufficient differentiation among them to effectively model samples from different unknown classes. In our experiments, we observed that some NPs tend to converge, collapsing to the same point in the unknown space, resulting in underutilization of the NPs.We incrementally increase the hyperparameter 𝑁 (the number of generated NPs) from 1 to 8 and record the changes in model performance with (w) and without (wo) the SA module. Simultaneously, we track the number of NPs that are utilized at least once during the entire testing process. The results, depicted in Figure 9 , reveal that without the SA module, the model can effectively use up to 3 NP generators. With the SA module, the model can fully utilize each generator, and the performance improves with the growth of 𝑁 until 𝑁 = 5, where the best model performance is achieved.Some models attempt to address the issue of collapsing by incorporating an equipartition constraint [25, 36] , which introduces a fixed-sized feature space for prototype assignment. However, in our experiments, we found that this approach fails to resolve the collapsing problem in the FSOR task. In both the DNPG model and the model without the SA module (DNPG -SA), we applied the When the SA module is not used, only a maximum of 3 NPs will actually be used (blue dashed line), after using the SA module, each NP can be used (red dashed line), and the effect is further improved (the red line continues to rise). equipartition constraint, similar to the method outlined in [25, 36] , to restrict the number of negative prototypes matching with samples. The results, as shown in Table 5 , indicate that adding the equipartition constraint negatively affects the model's performance on the FSOR task. In FSOR task, forcing each negative prototype to match in every episode is not suitable because the composition of known and unknown classes continuously varies across episodes. In certain cases, the unknown classes may consist of a similar set of classes, such as "Dalmatian" and "Alaskan Malamute." In such instances, it is more appropriate to allow the same negative prototype to represent them instead of forcing the use of a completely dissimilar negative prototype. This issue does not arise in our SA module, as we enable automatic matching between prototypes and negative prototypes, providing greater flexibility. In this paper, we investigate the task of few-shot open-set recognition. To generate diversified negative prototypes for identifying unknown class samples, we propose the DNPG model. DNPG decouples the known class sample information from the generation of negative prototypes. Instead, DNPG transfers the information learned from the unknown representation on the base classes to generate negative prototypes corresponding to novel classes. This approach ensures that the generated negative prototypes do not contain excessive known class sample information, thereby covering a broader range of unknown spaces. To ensure that each negative prototype has a certain performance and models the unknown space from different directions, we introduce the SA module, which guaranteed that multiple NPs will not collapse to the same point in the unknown space. Extensive experiments validate the effectiveness of our approach.  
: 
 rare disease diagnosis are typically small, and new users in recommendation systems have limited records. Secondly, models must contend with unknown data that falls outside the scope of training classes, such as online shopping platforms needing to identify and update new user-uploaded products.To address these challenges, Few-Shot Open-Set Recognition (FSOR) [10, 15, 17, 22, 31, 35] has emerged as a critical task. FSOR demands that a model utilize minimal training data from the support set to effectively recognize the query set, which involves classifying known-class samples and identifying unknown-class samples. This can be viewed as an N+1 classification problem (add an "Unknown" class), as depicted in Figure 1 . The FSOR model undergoes pretraining on base classes with ample data, which do not overlap with novel classes, before transitioning to the novel-class stage.Since the unknown class can be understood as an extra class against known classes, a common strategy is to learn a representation for this unknown class, referred to as the negative prototype (NP), and perform an N+1 classification task using the prototypes of both known and unknown classes [6] . Existing methods [15, 31, 41] typically generate NPs based on known-class data, as depicted in Figure 2 (a). However, this approach inherently limits the diversity of the NPs, as they tend to retain characteristics of the known classes. For instance, an NP generated for the "Dalmatian" class might resemble a generic dog with altered features, failing to represent the vast diversity of the unknown space. The unknown space is theoretically infinite, encompassing a wide range of samples that cannot be directly derived from known classes, such as a guitar being an unknown sample relative to the "Dalmatian" class.To overcome this limitation, we propose a novel approach to generate diversified NPs by learning unknowns from unknowns, thus alleviating the constraints imposed by known-class information. As illustrated in Figure 2 However, a challenge arises as the open weights, derived from base classes, remain static during the novel-class stage. This can lead to the generated NPs collapsing into a single point, reducing their effectiveness. To address this, we further introduce the Swap Alignment (SA) module, which ensures that NPs model distinct unknown spaces for different novel-class data, thereby preserving their diversity. During the base-class stage, we sample a set of nonoverlapping base classes as pseudo-unknown classes and minimize the distance between the generated NPs and these pseudo-unknown classes. Simultaneously, known class samples and pseudo-unknown class samples together form the swap tasks, where, at the prototype level, the distances between NPs and the pseudo-unknown class samples are further reduced. This approach ensures that each generated NP is distinct from all current known-class representations, preventing collapse and enabling adaptation to the current known classes.In summary, our main contributions are as follows.• We propose a novel FSOR method, the Diversified Negative Prototypes Generator (DNPG), to generate diversified negative prototypes by learning unknowns from unknowns, using the inverse representation of base classes. • We introduce the Swap Alignment module to prevent NP collapse and enhance adaptation to novel-class data, further increasing the diversity of NPs. • Extensive experiments have been carried out on multiple widely used FSOR datasets to prove that our DNPG model is superior to the current state-of-the-art methods. 
 
 Appendix:  


Decision: This work proposes the Diversified Negative Prototypes Generator (DNPG), to generate diversified negative prototypes using open weights from the unknown space while previous methods usually generate negative prototypes based on known class data. The idea is novel and sound, with the potential to attract interest from the open-set community. All the reviewers are satisfied with the author rebuttal. Despite that, the authors are encouraged to incorporate the reviewers' feedbacks into the revised version, and provide detailed computational complexity of their proposed model and the new components.

Reviews: review: paper_summary: paper_summary: This paper delineates a novel approach (termed DNPG) to Few-Shot Open-Set Recognition (FSOR), where a model needs to recognize known classes and identify unknown classes with limited labeled data. The paper claims that existing approaches mainly generate negative prototypes based on known class data, which limits their representation capability. In addressing this issue, the DNPG leverages the unknown space information learned from base classes to create more representative negative prototypes for novel classes.
strength_weakness: strength_weakness: S1. The paper's significance is underscored by the proposed DNPG, which exhibits an amalgamation of the diversified multiple negative prototype generator and swap alignment. 

S2. The clarity of the manuscript enhances accessibility for readers, facilitating a straightforward understanding of the proposed approach.

S3. The evaluation is comprehensive, with comparisons to baseline methods providing a compelling demonstration of the effectiveness of the DNPG model.
limitations: limitations: W1. The manuscript could explicate the GCN in the SA module, a factor which could be pivotal for understanding the proposed SA, e.g., its motivation and interpretation.

W2. The computational complexity of the DNPG model, particularly the attention mechanism, which could be a concern for large-scale datasets, is not discussed in the manuscript. 

W3. The paper could further explain and analyze the results in Figure 5.

Typos: 

- Line 284: fynction

- $d$ in Eq. (7).
paper_summary: paper_summary: The paper presents a novel approach, Diversified Negative Prototypes Generator (DNPG), for Few-Shot Open-Set Recognition (FSOR), addressing the challenge of recognizing both known and unknown classes with limited labeled data. The authors propose a method that leverages the unknown space information learned from base classes to generate more representative negative prototypes for novel classes. By pre-training on the base classes to learn the unknown space representation, and utilizing inter-class relationships in the meta-learning process to construct negative prototypes for novel classes, the proposed DNPG model aims to cover a broader unknown space, achieving state-of-the-art performance on standard FSOR datasets.
strength_weakness: strength_weakness: 1. The paper discusses the experimental validation of the model, using MiniImageNet and CIFAR-FS datasets, and evaluates the model's performance in comparison with existing methods. The authors demonstrate the effectiveness of the DNPG model in learning negative prototypes (NPs) with minimal known class information, simulating a more diverse unknown space. The model's capability to adapt to varying unknown class data is highlighted through comparative analysis with other FSOR models. Additionally, the paper details the functionality and performance enhancement brought by the introduced Swap Alignment (SA) module, ensuring adequate diversity among multiple NPs and addressing convergence issues.

2. The paper's theoretical foundation is well-developed. The use of open weights to directly generate negative prototypes from the unknown space is a unique and promising approach.

3. The paper effectively demonstrates the effectiveness of the DNPG model through comprehensive experimentation. The use of PCA for visualization and analysis of the generated negative prototypes adds depth to the experimental design.

4. The paper provides a strong comparative analysis between the DNPG model and existing methods, such as ATTG and GEL, showcasing the superiority of the proposed approach in simulating a broader unknown space and adapting to various unknown class data compositions.

5. The proposed Swap Alignment (SA) module effectively addresses the diversity and adaptability of the generated negative prototypes, contributing to a significant performance improvement. The thorough discussion and validation of this component add substantial value to the paper.
limitations: limitations: 1. While the experimental results on MiniImageNet and CIFAR-FS datasets showcase the effectiveness of the DNPG model, the paper could benefit from validation on additional datasets to demonstrate its generalizability across diverse data distributions and characteristics.

2. The paper lacks a detailed error analysis, which could provide insights into specific failure cases and the model's limitations in certain scenarios. A more in-depth exploration of model errors would contribute to a nuanced understanding of the DNPG model's strengths and weaknesses.

3. The paper does not extensively discuss the computational complexity and efficiency of the proposed approach, including training time, memory usage, and inference speed. Providing insights into the computational requirements would offer a more comprehensive evaluation of the practical viability of the DNPG model.

4. The paper does not delve deeply into the sensitivity of the proposed DNPG model to its hyperparameters. A comprehensive analysis of the model's performance across a range of hyperparameter settings would enhance the understanding of its robustness and stability under different configurations.
paper_summary: paper_summary: The paper proposes a modification of [15], a metric-based meta-learning framework for FOSR based on a negative prototype. The novel components of the proposed method include diversification of negative prototypes and a swap alignment module to sample most effective negative prototypes.
strength_weakness: strength_weakness: - The proposed method achieves the SOTA result.
limitations: limitations: - The proposed method is complicated, but its superiority over the current SOTA models is only marginal. Due to its complicated aspects, it is hard to know what purpose each component of the proposed method serves, and what it affects
 - Comparison to GEL may not be so accurate since its performance result was merely taken from the reference. In this aspect, the small improvement over GEL may be due to random seed difference. And in some cases, GEL outperforms the proposed method. 
 - The training/inference computational cost is not discussed.
 - Does the trend in Fig. 9 hold for TieredImageNet as well? Namely, is N=5 the best result?
 - What is the final training loss, and what score is used for open-set recognition in the inference stage?
paper_summary: paper_summary: This paper proposes a novel approach, termed Diversified Negative Prototypes Generator (DNPG), for Few-shot Open-set Recognition (FSOR). DNPG adopts the principle of "learning unknowns from unknowns". It leverages the unknown space information learned from base classes to generate more representative negative prototypes for novel classes. Experiments demonstrate the superior performance of the proposed method. The main contributions of this paper are:

- It proposes a novel FSOR method, the Diversified Negative Prototypes Generator (DNPG), to generate diversified negative prototypes by learning unknowns from unknowns, using the inverse representation of base classes.
- It introduces the Swap Alignment module to prevent NP collapse and enhance adaptation to novel-class data, further increasing the diversity of NPs.
strength_weakness: strength_weakness: - The problem studied in this paper is interesting and valuable.
- This paper provides some novel perspectives for FSOR.
- This paper is well written and in good sharp, which is easy to follow.
- The experimental results are promising.
limitations: limitations: - In Section 5.3, the authors only verify the contribution of each component on the MiniImageNet dataset. What about other datasets?
- Additionally, in Section 5.4, the authors only add the equipartition constraint (EC) to the DNPG model on the MiniImageNet dataset. What about other datasets?