Learning with Analogical Reasoning for Robust Few-Shot Learning

Authors: Hailong Zhu, Jun He, Xun Yang, Xueliang Liu, Richang Hong
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: few-shot learning, analogical reasoning, noisy label learning, contrastive learning, transformer

Abstract:
Few-shot learning (FSL) is challenging due to limited support data for model training. The situation is much worse when the support data is contaminated with noise. To address this issue, we propose a novel \textbf{T}ransformer-based \textbf{A}nalogical \textbf{R}easoning model for \textbf{N}oisy \textbf{F}ew-\textbf{S}hot learning (TarNFS), by mimicing the human's ability of learning by analogy. Concretely, we assume the existence of a large human cultivated or AI-powered knowledge base, and hypothesize that similar concepts in the knowledge base are visually similar in the latent space as well. Then we design a transformer-based analogical reasoning model to utilize inter-concept connections among these concepts, aiming to build robust and discriminative classification boundaries. In addition, we propose a task-level contrastive learning to analogically learn from negative tasks to facilitate training with noisy tasks. Experiments demonstrate that our TarNFS enables more effective learning from limited and imperfect data. It not only improves the generalization ability of FSL in different noisy settings but also achieves competitive performance in the common clean FSL settings. Code is publicly available \href{https://anonymous.4open.science/r/iclr2088}{here}.

Decision: reject
Reviewer scores: 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730425206314

thread_type:
official_review

<thread_object>
creator: reviewer_gftz

<content>
summary:
This paper proposes a Transformer-based Analogical Reasoning model, TarNFS, designed for robust few-shot learning (FSL) in noisy environments. It leverages analogical reasoning to improve classification by building connections among categories within a semantic knowledge base, like WordNet, and introduces task-level contrastive learning to separate distinct tasks further. The proposed approach is evaluated on MiniImageNet and TieredImageNet, showing improvements over previous FSL methods, particularly under noisy conditions, and demonstrating competitive performance on standard clean data.

strengths:
- The paper is well-organized, with a logical progression from problem formulation to methodology and experimental evaluation, making it accessible and easy to follow.

- The experiments cover a variety of settings, including different noise levels, ablation studies, and comparisons with several FSL baselines, providing a solid foundation for evaluating the model's effectiveness.

- TarNFS demonstrates advantages in handling noisy data, a challenging aspect of FSL, particularly through its strategy of refining noisy prototypes using analogical knowledge.

weaknesses:
- While MiniImageNet and TieredImageNet are commonly used benchmarks, the model’s real-world applicability could be tested on additional datasets, e.g. Meta-dataset.
- The model relies on an existing knowledge base (WordNet), which may limit its applicability in domains lacking such resources or where these resources are incomplete.
- The Transformer-based architecture and contrastive learning process, especially with Bi-LSTM in the task-level contrastive learning, may require considerable computational resources, which are not addressed in terms of efficiency or scalability.
- The model is tailored to the structure of WordNet for analogy-based learning. Adapting this method for domains with different types of semantic structures could present challenges, and further explanation of this adaptation process is warranted.

questions:
Can the proposed method outperforms the latest Noisy FSL method DETA [1]?

[1] DETA: denoised task-adaptation for few-shot learning, ICCV 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730590288582

thread_type:
official_review

<thread_object>
creator: reviewer_pjbs

<content>
summary:
This paper proposes TarNFS, a Transformer-based analogical reasoning model to improve Few-Shot Learning (FSL) in noisy environments. It enhances class prototypes using a knowledge base with multi-head self-attention and introduces task-level contrastive learning to improve generalization under noisy conditions.

strengths:
1.	The proposed analogical reasoning mechanism effectively incorporates contextual information from a knowledge base to overcome the limitations of noisy support data.

2.	The combination of multi-head self-attention and a large-scale knowledge base results in efficient enhancement of class prototypes.

3.	The inclusion of task-level contrastive learning provides additional regularization, significantly improving model generalization.

weaknesses:
1.	The details of knowledge base construction and selection for analogical reasoning are insufficient, particularly regarding applicability in different tasks.

2.	While task-level contrastive learning enhances generalization, the computational complexity is high, with no discussion on training time and efficiency.

3.	Scalability with larger or different types of knowledge bases is not discussed, which may limit the generalizability of the approach.

questions:
1.	During the construction of the query set using mixup, does the correlation between the sampled image and the combined augmented image affect the model's performance and stability? For instance, do cases where they belong to the same class or are completely unrelated impact the outcome?

2.	Under time constraints, does the computational cost of this construction method lead to a significant performance improvement in the model?

3.	How sensitive is the model's performance to the parameters used in the query set construction method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622165518

thread_type:
official_review

<thread_object>
creator: reviewer_xzn7

<content>
summary:
This paper addresses the challenge of Few-shot Learning with contaminated support data. To tackle this issue, the authors introduce a learning paradigm inspired by analogical reasoning. Specifically, the paper proposes a new approach to model inter-concept connections, which is then leveraged to improve inference in the presence of noisy support data. Additionally, the paper introduces a task-level contrastive learning strategy to mitigate the impact of noisy labels. To validate the effectiveness of the proposed method, the authors conduct experiments on several widely-used datasets, including MiniImageNet and TieredImageNet. The results demonstrate a significant enhancement in performance compared to existing methods.


=========================================================================
The rebuttal has addressed my concerns. I choose to increase my score.

strengths:
The concept of employing semantic and analogical reasoning to improve the performance of few-shot learning with noisy labels appears promising. The writing and presentation of the methods in this paper are clear and well-articulated.
This paper conducts a thorough analysis of the method using two datasets to demonstrate the contributions of each component and to compare performance across different benchmarks.
The results obtained from the two datasets indicate a significant enhancement in performance of the proposed method compared to existing approaches.

weaknesses:
While the author presents a detailed analysis using two datasets, I am concerned about their similarity. It would be beneficial if additional results could be demonstrated using other datasets, such as CUB and CIFAR-FS, to further validate the robustness of the method.
The methodology would be more convincing if it were also validated on a larger dataset, such as the Meta-Dataset. This would help in assessing its scalability and effectiveness across a broader range of learning scenarios.
The training procedure is multi-staged and could pose challenges in terms of reproducibility. Unfortunately, the author does not provide the code, which would be extremely helpful for those attempting to replicate the study's results.

questions:
From Table 4, I noted that the proposed method outperforms the oracle under a 20% noise setting. Could the author provide examples of several pairs of support and query instances where the oracle fails to classify correctly, but the proposed TarNFS method succeeds? This would offer deeper insights into the strengths of the method.
I observe that the TCL module exhibits a larger margin of improvement compared to AR. Could the author present an analysis that isolates the impact of using the TCL module alone? This comparison would help clarify the individual contribution of the TCL module to the overall performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733222874490

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
To validate the effectiveness of our method on more datasets, we conduct extra experiments on CIFAR-FS and CUB-200. We also tried different learners (like Conv4-64, ResNet12, ResNet18) to show the versatility of our method. Experiments results are as follows,

----

### **Results on MiniImageNet**

On MiniImageNet, we use the human curated knowledge base WordNet to discover concept connections for robust few-shot learning. 

> FSL with symmetric label swap noise on miniImagenet dataset (base model is Conv4-64).
>| Model \ Nois Proportion |  0% | 20% | 40% | 60% | 
>| ----------------------------| ----  | ----   |  ----  | -----  | 
>| MAML[1] | 63.25 | 53.28| 42.58 | 31.01 |
>| ProtoNet[2] | 68.27 | 62.43 | 51.41 | 38.33 | 
>| TraNFS-3[3] | 68.53 | 65.08 | 56.65 | 42.60 |
>| **DETA**[4] | **67.02** | **62.42** | **52.50** | **39.19** | 
>| TarNFS (ours) | 72.86 | 68.44 | 61.35 | 52.17 |

>Ablation study on MiniImageNet (base model is Conv4-64)
>| PN | AR | TCL |  0% | 20% | 40% | 60% | 
>| ----| ----| ----- | ----  | ----   |  ----  | -----  |
>| $\surd$| | | 70.16 | 64.36 | 54.13 | 40.25 | 
>| $\surd$ | $\surd$|  | 71.27| 66.78 | 60.06 | 51.34 | 
>| $\surd$ | $\surd$| $\surd$ | 72.86 | 68.44 | 61.35 | 52.17 | 

> FSL with symmetric label swap noise on miniImagenet dataset (base model is ResNet12).
>| Model \ Nois Proportion |  0% | 20% | 40% | 60% | 
>| ----------------------------| ----  | ----   |  ----  | -----  | 
>| PRWN[6] | 73.83 | 67.91| 47.32 | - |
>| RNNP[7] | 65.88 | 64.78| 50.62 | - |
>| PapNet[8] | 72.86 | 67.09 | 50.21 | - | 
>| IDEAL[5] | 75.90 | 71.36 | 57.35 | -
>|   DETA[4] | 81.67 | 76.58 | 65.13 | 47.60 | 
>| TarNFS (ours) | 82.09 | 76.69 | 66.49 | 51.97 |

----

### **Results on CIFAR-FS**

CIFAR-FS contains concepts that are not in WordNet. To cope with this, we utilize CLIP to build an AI-powered knowledge base in which the connections between two concepts are taken as the cosine similarities between them. 

> FSL with symmetric label swap noise and paired label swap noise on CIFAR-FS dataset (base model is ResNet12).
>| Model \ Nois Proportion |  0% | 20% | 40% | 60% | 40%(paired)|
>| ----------------------------| ----  | ----   |  ----  | -----  | ------------- |
>|         Oracle                     | 79.99 | 78.12 | 75.32 | 69.20 | 75.32 | 
>|   ProtoNet | 79.99 | 74.40 | 61.87 | 44.43 | 55.91|
>| TarNFS (ours) | 81.17 | 76.59 | 69.72 | 57.68 | 66.02 | 

>Ablation study on CIFAR-FS (base model is ResNet12)
>| PN | AR | TCL |  0% | 20% | 40% | 60% | 
>| ----| ----| ----- | ----  | ----   |  ----  | -----  |
>| $\surd$| | | 79.99 | 74.40 | 61.87 | 44.43 | 
>| $\surd$ | $\surd$|  | 80.37| 74.54 | 65.81 | 51.12 | 
>| $\surd$ | $\surd$| $\surd$ | 81.17 | 76.59 | 69.72 | 57.68 | 

----

### **Results on CUB200**

As there is no existing knowledge base corresponding to CUB200, we build an AI-powered one like that on CIFAR-FS. We use ResNet18 as the learner on this dataset.

> FSL with symmetric label swap noise and paired label swap noise on CUB200 dataset (base model is ResNet18).
>| Model \ Nois Proportion |  0% | 20% | 40% | 60% | 40%(paired)|
>| ----------------------------| ----  | ----   |  ----  | -----  | ------------- |
>|         URL[9]                     | 84.11 | 80.79 | 71.01 | 52.82 | 64.80 | 
>|   DETA[4] | 84.77 | 81.12 | 71.61 | 54.11 | 65.09|
>| TarNFS (ours) | 93.53 | 91.89 | 86.68 | 74.80 | 81.53 | 

----

In all the experiments above, we omit 95\% confidence intervals for brevity.

We believe that these experiments can thoroughly validate the effectiveness of our proposed method and demonstrate the generalization of our work to other datasets and learners. 

We made a revision to include part of the experiments so that the revision can address the reviewers' concerns properly.

----

[1] Finn et al. "Model-agnostic meta-learning for fast adaptation of deep networks." ICML, 2017.

[2] Snell et al. "Prototypical networks for few-shot learning." NeurIPS, 2017.

[3] Liang et al. "Few-shot learning with noisy labels." CVPR, 2022.

[4] Zhang et al. "Deta: Denoised task adaptation for few-shot learning." ICCV, 2023.

[5] An et al. "From instance to metric calibration: A unified framework for open-world few-shot learning." TPAMI, 2023.

[6] Bertinetto et al. "Meta-learning with differentiable closed-form solvers." arXiv preprint arXiv:1805.08136 (2018).

[7] Mazumder et al. "Rnnp: A robust few-shot learning approach." WACV. 2021.

[8] Lu et al. "Robust few-shot learning for user-provided data." TNNLS, 2020.

[9] Li et al. "Universal representation learning from multiple domains for few-shot classification." ICCV, 2021.
</content>
</thread_object>

</thread>


Metareview: