Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper conducts experiments which show that classification boundary in a few-shot setting is very sensitive to the position of support samples (few examples seen during evaluation) if they are in the vicinity of the data centroid (referred to as task centroid). Hence, a feature transformation named Task Centroid Projection Removing (TCPR) is proposed to remove the component of features along the direction of approximated task centroid. Experimental evaluation is conducted on several datasets.
strengths_and_weaknesses: Strengths:
The paper motivates the problem well with the “Lion” and “Dalmatian” example which highlights the problem with existing techniques. Later on, the paper addresses this problem with a simple solution by picking nearest samples from the base class and then picking a vector which points in the direction along the k nearest points in the base class, along a direction away from the class mean. Extensive experiments show the effectiveness of the method.

Weakness:
Some of the terminology being used in the paper is a bit non-intuitive to understand and this being coupled with incorrect sentences makes the paper hard to read.
questions: Is the "Removing the projection to the task centroid" similar to VLAD/Fisher Vector encodings which have previously been used in image classification? Is it possible to distinguish with these methods?

[1] https://openaccess.thecvf.com/content_cvpr_2015/papers/Xu_A_Discriminative_CNN_2015_CVPR_paper.pdf This was used for few shot event recognition.

sentence correction:

99: DCM [36] using support data to improve the evaluation performance calibrates the distribution of novel samples to approach zero centered mean and unit standard deviation . 

using support data to improve the evaluation performance calibrates the distribution  -> uses support data to improve the performance by calibrating the distribution

101: our method has different motivation to reduce the classification bias when the support samples in the vicinity of local centroid.

Not sure what this means
limitations: yes, they are addressed
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new technique, Task Centroid Projection Removing (TCPR) for few-shot classification. The paper highlights the phenomenon of the distance of samples from a task centroid affecting task performance. The paper then builds on this insight to create the TCPR technique. 
strengths_and_weaknesses: ___Strengths___: The paper is attacking a significant problem in machine learning and is presenting a novel technique to address that significant problem. Learning with less labels, which includes few-shot learning, is one of the greatest problems hindering machine learning. So, this paper is directly addressing a very significant problem. The technique proposed as well as the insight about support samples’ positions in the embedded space is rather novel and straightforward. The idea of pre-training bias affecting subsequent few-shot classification is novel and also reflects some of the known wisdom regarding domain adaptation in machine learning. The use of base set in constructing the task centroid is a clever and straightforward means of controlling the variability that can arise in sampling for few-shot classification. 

The paper also does a thorough empirical evaluation, including an ablation study with different models and hyperparameters. The thoroughness of this study combined with the released code greatly inspires the reader to try the proposed technique on their own work.

___Weaknesses___: The paper has a few weaknesses. First, there are some minor clarity issues. The placement of figures is wonky in that Table 3 comes before Table 2. And, while this is an acknowledged limitation, the setting of $k$ for the number of nearest neighbors really needs more guidelines for practical use. 

questions: Why is cosine distance used for all similarity measurements? Could any distance metric be used?

On line 233, what are ‘cosine classifiers’? Could you give a more thorough explanation of how the final classification was done for the query set.
limitations: The authors have addressed the limitations of their work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper claims to be the first to investigate bias of the supports that causes low accuracy in few-shot learning. Specifically, when the supports are too close to the centroid of a task, the quality of the decision boundaries would be unstable and hence a lower accuracy. They propose a framework to remove such bias by firstly estimating the centroid with base and support features from a pre-trained feature extractor and removing the components that lie in the same direction as the centroid, pushing away the novel features, and thus mitigating the bias. With the experiments on several benchmarks, they showed the effectiveness of the method.
strengths_and_weaknesses: Strengths

1.	The idea is intuitive and easy to understand and follow, which they also show the effectiveness by applying the method on different pre-trained few-shot models.
2.	They showed SoTA performance on different benchmarks.
3.	A set of ablation studies are performed on the design choices.

Weaknesses

1.	No analysis on failure cases, if any. Discussion on limitations is limited. 

questions: 1.	The assumption is novel features tend to mingle with base features, and one could find K nearest of them for task centroid estimation. Any guarantees on this? What if novel features lie in a sparse region where the closest base features are far? Would it cause instability for centroid estimation? More exploration and evidence on this would be greatly appreciated and would bolster the work.

2.	The simulation experiments are in 2-d, difference in high dimensional space?

3.	I would imagine in some cases, removing the projection to the centroid could potentially cause performance degradation? For instance, what if supports from two classes lie in the upper sphere while supports for the other two lie in the other sphere? Then, in this case, initially we could use the direction along the centroid for differentiation, and yet after projection, we might not be able to?

limitations: No discussion on limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the sampling bias issue in few-shot learning and identifies its connection to task centroids. To mitigate the sampling bias, this paper proposes to perform Task Centroid Projection Removing (TCPR) before training the classifier for the test task. When performing TCPR, it approximates the task centroid by using top K data samples in base classes. Experimental results show the proposed method improves the classification accuracy for different pre-trained feature extractors.
strengths_and_weaknesses: Originality: 
The studies about the connection between sampling bias and task centroid and the proposed TCPR method are novel.
	
Quality:
The proposed TCPR is technically sound. 
The paper tries to support the claim about the connection between sampling bias and task centroid through toy cases and experiments. But the toy case in Sec 3.3 is not a good toy case, because it does not match with the cases where TCPR is applicable.
Another issue is computation cost. The value of k is very large. The time cost to get top k base samples may be high. Moreover, this method requires access to all base samples, which may be not practical.
	

Significance: 
The sampling bias is an important issue in few-shot learning. This paper studies the connection between sampling bias and task centroid and proposes TCPR to mitigate it. Applying TCPR on various pre-trained feature extractors, significant performance improvements are achieved.

Clarity:
The paper is not difficult to understand. It clearly presents the motivation and the method and provides sufficient technique and experiment details. But the writing can be further improved. 

----
After rebuttal:
The response from the authors addresses my questions. I still hold some minor concerns about the failure of TCPR on the toy case, as the authors did not provide a revised version of the paper, and also about the large value of k.
I keep my score weak accept. 



questions: 1. The concern about the large value of k. Could the authors provide experiment results about the running time cost of obtaining top k base samples and training the classifier?
2. The toy case in Sec 3.3. TCPR cannot be applied in this case. I think it would be better to project the data into a 3-d hypersphere and perform TCPR to see how TCPR helps to improve the performance. I think TCPR would work for some cases and fail for other cases. 
3. TCPR is somehow similar to DC[43] as both methods try to improve the performance by using the information of base samples. I think it would be better to provide more discussion about DC in related works and provide more experiment results of DC on the same pre-trained feature extractors as TCPR, e.g. Baseline++, Inv-Equ, S2M2, S2M2*. Besides, is it possible to retrieve top k base classes prototypes (like DC) instead of top k base samples, so that not all base class samples are required?
4. About Experiment in Table 3: It would be desirable to provide the distance between the approximation of the different task centroids and the real task centroid. In addition, the results of the 5-shot experiments should be provided in Table 3 for a better understanding of the improvement of c_task on the 5-shot experiments.
limitations: The paper discusses the limitation about selection of k.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a novel few-shot learning method of task centroid projection removing (TCPR). The proposed TCPR aims to resolve one of the limitations of few-shot learning, called the sampling bias problem. By adjusting the classification boundary via moving the data centroids by removing some components of features, TCPR can improve the few-shot classification accuracies for various datasets, feature extractors, and base few-shot classification algorithms.    

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
After Rebuttal: I appreciate the authors for kindly addressing my concerns and questions. Most of my concerns are resolved. As a result, I keep my original score.
strengths_and_weaknesses: Strengths

1. Idea of approximating task centroid with the help of base class data is interesting and powerful.  
2. The concept of utilizing the task centroid to resolve sampling bias problem is intuitive and powerful, and it can be easiliy applied to various FSL techniques. Experimental results show that the proposed TCPR is able to improve the performances of various few-shot learning methods.
3. Analysis results on sampling bias problem would be helpful for further research on sampling bias problem. 

Weakness

1. It seems the process of sampling the most related samples from the base class samples seems computationally inefficient. Since there are many base class data and N_base is large, it seems that computing cosine similarity and choosing top-K samples from base data require many computations. 
2. The task centroid has to be computed anew for the given task, and we need to keep the entire base data, not just the trained model. For me, it seems that keeping all the base class samples for the approximation of task centroid seems somewhat unreal. 
3. I am not sure whether the comparing TCPR-extended few-shot learning with the base few-shot learning model. In Figure 7, it seems that TCPR requires the large k to obtain meaningful gain, which means TCPR utilizes many extra data, although the extra data come from the base classes. 

questions: 1. Related to weakness 2, can we utilize the base class centroids instead of the base class data? Since there exist many data for the base classes, we can easily compute the true class centroids for the base classes. Isn’t it more realistic to store the base class centroids, not the entire base class data? How does this affect performance?
2. The process of removing the projection to the task centroid seems to make the features orthogonal to the task centroid and remove the effect of the dimension of task-centroid. Is there any other way to remove the dimension of task-centroid? For example, [1] utilizes the linear nulling of the vector to remove the effect of specific dimensions. 

[1] Yoon et al. TapNet: Neural Network Augmented with Task-Adaptive Projection for Few-Shot Learning, ICML 2019.

limitations: 1. Sampling process for computing task centroid may be computationally inefficient.
2. To obtain the gain of TCPR, we need to store some extra base data, which can be unreal condition.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a post-processing technique to improve the quality of prototypes used in few-shot learning (FSL). The main idea is to remove a bias, by first identifying a large set of neighbors from the base samples (training data used in designing the model) and performing a weighted averaging to identify the origin of decision space (my interpretation). Once the center is identified, the support and query samples are normalized according to the center (by first a weighted bias removal and then normalization). Experiments on various datasets show that the method helps in having a less biased decision space, hence improving the results. 
strengths_and_weaknesses: Strengths
-------------

- the paper is written very clearly.
- focusing on classification, the authors did a comprehensive study and contrast their algorithm on a broad set of experiments, including meta-dataset.

Weaknesses
-------------

- despite the fact that the authors explained the motivation behind the work (closeness to a center), I have a lot of difficulties convincing myself that the so-called task centroid is the source of the problem. Overall, I think one can scale samples, and distances to the task-center cannot be the only reason behind improvements here. I do understand that the solution works on the hyper-sphere so do not crusade me but my point is, the relative closeness cannot be the sole reason behind the improvement obtained in this work.

- the method is quite hefty in compute as it requires identifying k-neighbors of support samples, where k in practice can be in the range 10,000. As a matter of fact, Fig.7 suggests the algorithm underperforms on mini-imagenet if k <100.

- the problem studied is solely classification. 
questions: As for questions, I would like to ask

1- Can the method be used to address dense prediction FSL? if yes, how?

2- I am not convinced that the change of origin is the source of bias. Somehow I feel that the algorithm tries to use the base data to construct a better latent space. A natural question to ask is, can we consider the PCA space from the base samples as a way of denoising the decision space? That is, aside from the task-center, what will happen if the PCA space is also taken into account. Can you kindly comment on this?
limitations: It is not discussed but the method extensively uses training data towards its decisions so could potentially generate unfair and biased outputs
review_title: null

Meta-review: metareview: The paper investigates the bias of support points in few shot learning when they are too close to the task centroid. This can cause a drop in accuracy. It proposes a method to mitigate this by projecting out the task centroid direction. Initial reviews were split on the paper with three reviewers having a positive opinion while the other three had concerns. However the author response has addressed many of the reviewer concerns (also acknowledged by one of the reviewers). The paper makes a solid contribution to the few shot learning problem and will be a good addition to the program. 
confidence: Certain
Decision: Accept