Prototype antithesis for biological few-shot class-incremental learning

Authors: Binghao Liu, Han Yang, Fang Wan, Fei Gu
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Biological Recognition; Few-Shot Learning; Class-Incremental Learning; Prototype Antithesis

Abstract:
Deep learning has become essential in the biological species recognition task. However, a significant challenge is the ability to continuously learn new or mutated species with limited annotated samples. Since species within the same family typically share similar traits, distinguishing between new and existing (old) species during incremental learning often faces the issue of species confusion. This can result in "catastrophic forgetting" of old species and poor learning of new ones. To address this issue, we propose a Prototype Antithesis (PA) method, which leverages the hierarchical structures in biological taxa to reduce confusion between new and old species. PA operates in two steps: Residual Prototype Learning (RPL) and Residual Prototype Mixing (RPM). RPL enables the model to learn unique prototypes for each species alongside residual prototypes representing shared traits within families. RPM generates synthetic samples by blending features of new species with residual prototypes of old species, encouraging the model to focus on species-unique traits and minimize species confusion. By integrating RPL and RPM, the proposed PA method mitigates "catastrophic forgetting" while improving generalization to new species. Extensive experiments on CUB200, PlantVillage, and Tree-of-Life datasets demonstrate that PA significantly reduces inter-species confusion and achieves state-of-the-art performance, highlighting its potential for deep learning in biological data analysis.

Decision: accept (poster)
Reviewer scores: 3, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729325241021

thread_type:
official_review

<thread_object>
creator: reviewer_zao3

<content>
summary:
The article proposes a Prototype Antithesis (PA) method, which first learns species-unique and family-shared semantics of original classes. For new classes, it promotes learning by mixing family-shared features while preserving the model's ability to discriminate original classes. Experimental results validate the effectiveness of this method.

strengths:
1. The paper is well-organized.
2. The literature review is detailed and comprehensive.
3. The suite of experiments is admittedly comprehensive.

weaknesses:
1. There are three variable in cosin similarity calculation, please explain how the cosine similarity is specifically calculated in Eq 2 for  $\hat{Y}_{i}^r=cos(R_i, \theta_c, \theta_r)$ and $\hat{Y}_i^0 = cos(F_i^0; \theta_c, \theta_r)$.
2. How is the family-level label defined? Is this label provided by the dataset?
3. What is the number of features for $\theta_r$, and does this number change during training?
4. Why does mixing the residual prototypes of old species with new species features encourage the model to capture the unique characteristics of the new species?
5. Does the method require integrating residual features during the inference phase?

questions:
The technical details are unclear; please see the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730680232899

thread_type:
official_review

<thread_object>
creator: reviewer_kran

<content>
summary:
The authors present a novel approach to handle the few-shot class-incremental learning problem, specifically in species classification. The authors take advantage of the hierarchical taxonomic relationship between species and use their family label to transfer knowledge from existing species and new species. Their approach, named Prototype Antithesis (PA), employs Residual Prototype Learning (RPL) to learn unique prototypes per species and novel residual prototypes to represent shared traits between species in a family. To further improve the model’s discriminating ability, they utilize Residual Prototype Mixing to generate synthetic samples for data augmented training. They show SOTA performance on three biological datasets: CUB200, PlantVillage, and Tree-of-Life.

strengths:
- They show the generalization of their ability by evaluating on three biological image datasets.
- The authors present novel residual prototype learning and residual prototype mixing training methods that improve class separating and alleviate catastrophic forgetting.
- They present a theoretical analysis of their method, focusing on class-separation.
- An ablation study shows the quantitative effectiveness of their residual prototype learning and mixing methods.
- Their method shows stronger performance than others in the later stages of the incremental learning.

weaknesses:
- The feature decomposition strategy section is convoluted, so Table 2 doesn’t make sense to me. I understand that there are high response and secondary response regions being extracted, but I’m confused how they are used to obtain the values in Table 2.
- You have all of your model’s accuracies bolded in Table 3, but at several columns, your model is not the best (columns 0, 1, 2, and the PD). Please correct.
- The method is only evaluated on 100 classes in the tree-of-life dataset. This should be run multiple times to ensure the robustness across different sections of the dataset given its size.

questions:
- Could the author help clarify my confusion about the Feature decomposition strategy in section 5.2
- See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710918071

thread_type:
official_review

<thread_object>
creator: reviewer_p2dq

<content>
summary:
The work observes that a limitation of incremental learning is confusion with closely related species (like species from same family) and therefore makes use of family label of each species to overcome that. The method learns one family level prototypes representing shared traits, and species level prototypes representing species specific traits. In the initial learning stage prototypes are learned through a method introduced called Residual Prototype Learning and during incremental learning stages a method called Residual Prototype Mixing is introduced.

strengths:
1.	Extensive comparison with multiple relevant baselines for CUB200.
2.	Family information of any species is easy to obtain, so using that to improve incremental learning makes sense.

weaknesses:
1.	Basis for some assumptions made are not clearly explained. Please refer to the questions.
2.	Not every baseline is considered for all three datasets. No particular reason is provided for skipping several baselines. Particularly I believe NC-FSCIL is a necessary baseline since its performance is pretty close to PA without having to use family label. If the skipped methods are not suitable for Tree-Of-Life and Plant-Village, please explain.
3.	No discussion of a possible limitation: In RPM are we assuming no new species is introduced whose family is not seen during the initial learning phase. If so, that is possible limitation of the work and it has to be discussed.

questions:
1.	Line 274, how the assumption that prototype do not contain background information is made? Is it not possible that both family level and species level prototypes may contain background information.
2.	Line 275 and 276, how can it be assumed that the species level prototypes (mu_ck) contain both family-shared and species-unique information, while in the RPL (line 231 and 232) it is assumed they capture the most species unique traits.
3.	In section 4.3, it is not clear from Equation 3 how the delta of cosine similarity is optimized as mentioned in line 289. Is it happening implicitly when we optimize as in Equation 3.
4.	In section 5.3 Feature Distribution subsection, does “without PA” mean the baseline model from Table 1, if so, please mention it explicitly
5.	Throughout the main paper, I’m assuming the cosine similarity between three vectors as in Equation 2 and 3 means the summation (or average?) of cosine similarity of the feature vector with respect to the two prototypes, but it would be better to explicitly mention it.
6.	In table 3, PA’s performance is bolded. If the best performing method at each session can be bolded instead that would improve the readability of the table, making it easy to understand which method is doing better as I can see in Table 3, PA is not doing the best in all sessions (can also consider highlighting first, second and third best differently)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730835813709

thread_type:
official_review

<thread_object>
creator: reviewer_qsws

<content>
summary:
This paper studies "fine-grained" class-incremental learning in a few-shot setting. The authors argue that under such a setting, the model would easily suffer overfitting (to new classes) or forgetting (of old classes). They thus propose to leverage hierarchical-class information (e.g., family-species) to encourage knowledge sharing across species of the same family while learning discriminative information. The paper proposes two new learning strategies for this purpose: RPL and RPM. In three small-scale experiments, the proposed approach demonstrated improved performance.

strengths:
S1. The paper focuses on an interesting and challenging problem. 

S2. The paper points out several insights that future research on fine-grained incremental learning could leverage.

weaknesses:
W1. It is unclear or not intuitive why when new classes and old classes are similar (so have shearable information), incremental learning will lead to forgetting. 

W2. The technical/implementation details are not clear and not optimally designed. First, ResNet18 seems to be a too-weak backbone, especially for fine-grained problems. The authors may consider transformers (DINO, DINO-v2, BioCLIP visual encoder) or ResNet50 at least. Second, there is no information about whether the backbone is pre-trained, and if so, on what dataset. Third, it seems the feature backbone is only updated during the base training stage (Eq 1). If so, suppose the linear classifiers of the base (old) classes are frozen during continual learning, I see no reason why the old classes will be forgotten. Fourth, do the authors impose an orthogonal constraint in Line 270?

W3. My major concern is the approach itself. The design seems to be quite ad hoc without justifications. First, I'm not sure why computing the residual between the linear classifiers (or prototypes) and feature vectors makes sense. Please note that the prototypes can simply be the feature vectors times a scalar. Why does the residual contain information on "the secondary discriminative features likely representing traits shared across the family?" Second, why does adding the residual to feature vectors of the new species make sense? What do the new feature vectors encode? Third, the meanings of the decompositions and terms introduced in Lines 270 - 290 are not clear or justified. Finally, if the residual is added to features of the new species during training, how about the inference stage?

W4. The experiment details are missing; the experimental design can be improved. For example, no dataset statistics are provided. No information about how the authors obtained the family information. The PlantVillage dataset seems to be too small; the authors could consider using families in iNaturalist. No information about how the Tree of Life Dataset (with ~400K species) is subsampled. How do the authors ensure that all the families are seen during the base training time? Finally, experiments on no more than 200 species are a bit too small. 

=== Minor ===

W5. There are many missing references, for example, no references to the datasets in the introduction; no reference to "Inspired by human’s reference-based learning mechanism."

W6. The related work seems to miss one topic, generalized few-shot learning, such as "Generalized zero-and few-shot learning via aligned variational autoencoders, CVPR 2019." Will this line of work resolve the problem in Line 138?

questions:
Q1. What is the detail of the Feature decomposition strategy (Line 400)? CAM and Grad-CAM are methods to visualize a classifier. How did the authors use their saliency responses to get accuracy in Table 2?

Q2. BioCLIP is a foundation model pre-trained on over 400K species. Can the authors provide more details about the argument that "it struggles with continual learning and tend to misclassify closely related species?" (Line 126)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732021179634

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all reviewers for their time and positive evaluation of our work. Namely, our paper addresses "an interesting and challenging problem" (qswS) and introduces "novel training methods that improve class separating and alleviate catastrophic forgetting"(KRan). Our proposed method "makes sense to improve incremental learning"(P2DQ) and demonstrates "the generalization of their ability by evaluating on three biological image datasets"(KRan). Furthermore, our paper provides "a theoretical analysis of their method"(KRan) and includes "a detailed and comprehensive literature review"(Zao3), along with "a suite of experiments that is admittedly comprehensive"(Zao3).

We also sincerely thank all the reviewers for their insightful comments and suggestions. We have uploaded a revised version of the paper, incorporating the reviewers' helpful suggestions. The main changes are listed below (detailed in the individual responses):  
- Expanded references and additional discussion of related works.  
- Clarification of the inference stage process in detail.  
- Detailed explanation of the cosine similarity calculations in Eq. 1, Eq. 2, and Eq. 3.  
- Explanation of the differences between species prototypes learned during base training and incremental learning.  
- Additional experimental details, including whether the backbone is pretrained, robustness of results, and more dataset statistics.  
- Improvements to Fig. 3, Table 2, and Table 3 to enhance clarity and readability.  
- Inclusion of comparisons with more prior works, such as NC-FSCIL, on the PlantVillage and Tree-of-Life datasets.
   
To facilitate the review process, we have marked the revised parts in the paper in red and provided point-by-point responses to each reviewer's comments. We hope this revised version is clearer, more complete, and demonstrates improved theoretical justification and empirical validation.  

Once again, we extend our gratitude to the reviewers for their efforts and valuable feedback, which have greatly contributed to enhancing the quality of this paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732674964912

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Kindly Request for Final Reviewer Queries and Score Adjustments

comment:
We extend our heartfelt gratitude to the reviewers for their insightful comments and invaluable feedback, which have greatly contributed to improving our work. As the discussion period draws to a close, we warmly encourage any additional questions or requests for clarification. Should our responses have sufficiently addressed your concerns, we would sincerely appreciate your consideration in revising the scores and confidence levels accordingly. Thank you for your time, effort, and thoughtful engagement throughout this review process.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work studies the problem of few-shot class-incremental learning in the setting of biological species recognition. To solve the "catastrophic forgetting" problem, the authors designed a Prototype Antithesis method, which reduced the confusion between old and emerging species leveraging the hierarchical structures in biological taxa. This idea is interesting and novel. Besides, the experiments are extensive; on three large datasets the proposed method significantly reduced the inter-species confusion and achieved SOTA. In addition, there are still some minor issues, for example, lacking more explanations of the motivation and method design, which could still be enhanced. Overall, this work could be accepted.