Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a transfer learning method for few-shot regression/classification. In representation learning, it adapts the stochastic weight averaging (SWA) as a regularizer for learning more generalizable features. In fine-tuning phase (adaption to a new task in the meta-test phase), it treats the regression and classification differently. For regression a hierarchical Bayse is used while in classification, a conventional method is merged with temperature scaling. This work achieves comparable results to SOTA, however, they are major concerns that need to be addressed. 
main_review: There are several concerns that needs to be addressed, and I believe addressing these can improve the quality of this paper. 

 
 

- There are some typos in the main text, like Prot Net in 5.5. 

 
 

I have some major concerns about the opening remarks of this paper listed below: 

 
 

- In the introduction, the authors have mentioned that : “Despite the success of episodic meta-learning in few-shot learning tasks, they are slow to converge, prone to over-fitting, and tricky to implement (Antoniou et al., 2018 [1]).”  

I think this statement is not generally true, as these are the shortcoming of the MAML algorithm listed in [1], and most of them are addressed in this paper as MAML++. Also, as an example recent metric-based methods (ProtoNet, RelationNet, FT for cross-domain FSL [2]) does not suffer from these issues that much, at least when comparing to the transfer learning methods like Chen et al. [3] and Dhillon et al [4].  

For example, ProtoNet, has a relatively fast convergence and increasing the ways in meta-training is proved to resolve overfitting. There are also several recent extensions, and reqularizers for few-shot learning that address these problems well. 

 
 

- In the later part of the introduction, you also mentioned that your algorithm tries to address these shortcomings: “In this paper, we propose a new method that does not rely on meta-training to overcome the limitations of commonly used episodic meta-learning approaches in few-shot learning.”. But, there is no evaluation regarding faster convergence, preventing overfitting, and simpler implementation of your algorithm compared to recent episodic ones. 

 
 

- I also doubt the validity of this statement in the introduction: “recent studies on transfer learning (Chen et al., 2019; Tian et al., 2020) cast doubt on whether it is the episodic meta-learning algorithm or the learned representation that is responsible for fast adaption to new tasks”.  

Specifically, I am not so sure about differentiating episodic meta-learning and learning better features. As an example, Goldblum et al. [5] show that meta-learned features have a better generalization compared to conventionally trained networks with exactly the same structure. So, using the meta-learning algorithms leads to the better representation learning and these two are not to independent concepts. 

 
 

- What do you exactly mean by this?  

“Our method is the first one to achieve well-calibrated few-shot models by only fine-tuning probabilistic linear models in the meta-test phase, without any learning mechanisms related to the meta-training or representation learning phase.”  

This mechanism is the well-known solution in transfer learning methods which is firstly popularized by [3] for few-shot learning. Am I missing something here? 

 
 

Considering all these problems, I think the authors need to rewrite the introduction part and make it more reliable, considering the recent studies in few-shot learning. There are also some concerns for the Methodology part: 

 
 

- Based on my understanding, your representation learning part is similar to [3], and the only difference is the usage of SWA as post-processinf of the network parameters. Although it is mentioned that the purpose is to find a low-rank approximation, it is not clear that how do you finally deploy it in your algorithm. The algorithm is unclear and seems to be lost in the explanation of various concepts! 

Are you using a simple averaging (mentioned in equation (1)) as a regularizer and just describing that this implicitly steers the model towards low-rank representation? 

 
 

- For the fine-tuning in few-shot classification, it is not clear for me how logistic regression model fits within your framework.  

 
 

- I think adapting the temperature scaling for meta-test phase is the only contribution for fine-tuning few-shot classification. Is this new for few-shot learning to be considered as new algorithm? 

 
 

- Overall, I found it very hard to follow the methodology as in some cases, the main flow of the algorithm is lost due to overexplanation of other concepts. 

 
 

Regarding experiments: 

 
 

- The few-shot regression results are interesting, but the few-shot classification results are not fair, because some recent algorithms like MELR [6] are not included in the comparison. 

 
 
 
 
 

References:  

[1] How to train your MAML, ICLR 2018. 

[2] Cross-Domain Few-shot Classification via Learned Features-wise Transformation, ICLR 2020. 

[3] A closer look at few-shot classification, ICLR 2019. 

[4] A Baseline for Few-Shot Image Classification, ICLR 2020. 

[5] Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks, ICML 2020. 

[6] MELR: Meta-Learning via Modeling Episode-Level Relationships for Few-shot Learning, ICLR 2021. 

 

summary_of_the_review: In summary, the proposed algorithm seems interesting. However, there are some major problems in the opening remarks of the paper, the explanation of the algorithm is not clear enough, and the few-shot classification performance of the proposed method is not fairly compared with recent methods. These concerns need to be addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a few-shot learning method based on the reprehensive pre-train learning using stochastic weight averaging (SWA). The merits of the proposed work is that this proposed method can works for both few-shot regression and classification problems and both of them achieves better results compared with the other recent works as reported in the manuscript. And as claimed by the author, this is the first few-shot learning work can works for classification and regression.

main_review: This paper presents a few-shot learning method based on the reprehensive pre-train learning using stochastic weight averaging (SWA). The merits of the proposed work is that this proposed method can works for both few-shot regression and classification problems and both of them achieves better results compared with the other recent works as reported in the manuscript. And as claimed by the author, this is the first few-shot learning work can works for classification and regression.

1), It is well-known that SWA is an effective tool in weight space to obtain better reprehensive learning as it leads to a more flat minimum. As a result, it is not very supervising that SWA achieves better results in transfer learning for few-learning learning. Actually, SWA improves most applications as the author mentioned for meta-based approaches. Therefore, employing SWA for few-shot learning might not be a contribution that significant enough for an individual work. It will be more interesting to provide more analyse why SWA lead to the better performance compared with other solutions. Moreover, the flat loss surface contribute to the adaption is well discussed in some existing incremental few-shot learning research.

2), It is interesting that the authors use the theory for meta learning to guide the research for transfer learning based few-shot learning, i.e., to make the feature with lower-rank representation. The result shows that the performance is improved. However, the theoretical link is still not clear. Is the low-rank representation criterion general for all reprehensive learning to have better result? Or the SWA is best way to achieve the low-rank representation for few-shot learning? It might be suspected that all those learning methods resulting better performance will contribute to the low-rank representation, e.g., various pre-training losses, network structure, etc.

3), For the transfer learning based few-shot learning method, there will be a problem that by freezing the feature extraction, it is almost impossible to further adapt and generalize to cross-domain test environment if the feature difference between base classes and few-shot classes are significant. As a result, the evaluation performance will be very related to the dataset whether the few-shot/pretrain data are similar or not. In contrast, methods like MAML will adapt and finetune the feature extraction for the novel classes, which leads it may more difficult to train. However, its generalization capability will be stronger when the feature of few-shot classes are different with the pretrain base classes. As a result, to prove the effectiveness of the proposed methods, more ablation should be added with different setting, e.g., the few-shot classes close to the pretrain base classes, few-shot classes far away to the pretrain base classes, etc. In this way, the reader will be able to see the comparative advantages and limitations of the proposed method. Otherwise, it might not be proper to directly conclude the proposed methods is better than the existing methods.

4), Similar like SWA, Exponential Moving Average (EMA) has the similar better results than the directly trained weights, and both are approaches in the weight space. It will be reasonable and interesting to be an ablation study to see if EMA achieves the similar results in few-shot learning or if EMA also contributes to the low-rank representation.

5), Similarly, for other transfer learning based few-shot learning method such as Baseline++, it is straightforward and fully compatible to employ the SWA in its pertrain, the comparison should be analysed if MFRL better than (Baseline++ with SWA) to demonstrate the effectiveness of the MFRL.
summary_of_the_review: As mentioned in the Main Review above, the merits and concerns are pointed for reference and discussion. Generally, the main concern is on the originality as the SWA is an existing method, and the detail analysis on the mechanism how SWA specifically contribute for few-shot learning is not adequate.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The submission introduces a few-shot classification and regression approach called Meta-Free Representation Learning (MFRL). First, a representation is learned on the meta-training data: for regression, the model is trained on all available training regression tasks concurrently; for classification, the model is trained on the full-ways classification problem using all training classes. Then, stochastic weight averaging (SWA) is applied to the model by continuing training for a certain number of epochs and averaging the parameters obtained across those additional epochs.

For test tasks, the regression or classification layer is discarded, and a new output layer is trained while freezing the network weights. For regression, approximate inference is performed on a hierarchical Bayesian linear model. For classification, a logistic regression model is trained with L2 regularization and a temperature hyperparameter.

Experimental results are presented on the sine wave and head pose problems for regression, and on mini-ImageNet, tiered-ImageNet, CIFAR-FS, and FC100 for classification. The extent to which SWA encourages learning lower-rank representations (as hypothesized) is verified through visualizations of the normalized eigenvalues. Finally, calibration curves are shown for classification to demonstrate how MFRL's temperature scaling factor leads to better calibrated models.
main_review: The writing is easy to follow. The introduction provides a good motivation for the proposed approach and presents a thorough review of the existing literature. The idea of using SWA for few-shot classification is itself interesting and new as far as I can tell, and the empirical results look promising. The conjecture that it fosters lower-rank representations is supported by empirical observations.

However, after reading the paper, I'm left with the question: beyond the application of SWA to the few-shot learning setting, what is MFRL? One of the claimed contributions is that it "can handle both few-shot regression and classification", however different adaptation strategies are used for regression (hierarchical Bayesian linear model) and classification (L2-regularized logistic regression with a temperature coefficient). As far as I can tell, the aspects common to both settings are i) learning the representation on the training data using a single objective rather than episodic losses (which has already been explored in previous work), and ii) applying SWA. In fact, Algorithm 1 feels more like two algorithms combined with if-statements than a unique algorithm. I feel that the presentation would have been better and more straightforwardly characterized as an application of SWA to the few-shot learning setting, and I'd be interested in hearing the authors' opinion on this.

Additional questions/comments:

- Can the authors say a few words on hyperparameter selection using meta-validation data? In the cross-domain setting (e.g. mini-ImageNet -> CUB, or Meta-Dataset), is this a good selection strategy?
- I am surprised by how well MFRL without SWA performs for regression. Can the authors expand on what aspect of MFRL is responsible for its good performance? As I understand, it consists in training concurrently on all training regression tasks; how is it that it outperforms more sophisticated meta-learning approaches?
- In the classification setting, MFRL only really stands out when SWA is applied. Are there other aspects to MFRL that positively impact performance?
- The difference between MFRL and Baseline++ (one normalizes the features, and the other normalizes both the features and the classification weights) is easy to miss. Being such a small implementation difference, it also reinforces the idea that MFRL reduces to SWA, and naming it is not necessary.
- Presenting results on model calibration is good, but the paper feels lacking in details. Judging by the introduction, using a temperature parameter is sufficient to obtain good model calibration. How come? The submission mentions that "given a meta-test task, a new  top layer is fine-tuned with few-shot samples to obtain probabilistic models with well-calibrated uncertainty"; what about the fine-tuning procedure ensures calibration? Is some kind of early-stopping performed by looking at calibration metrics?
summary_of_the_review: The submission applies an existing idea (SWA) to the few-shot learning setting, which is interesting and shows promising results. However, there is a disconnect between the simplicity of the idea and how it's presented by the submission, and overall there aren't a whole lot of new insights to be gained beyond the observation that SWA works well for few-shot learning.

---

**POST-REBUTTAL**: The response addresses most of my concerns. I would like to see those clarifications integrated in the main text.
review_title: null

Meta-review: By the scores, this submission is quite borderline. This paper introduces stochastic weight averaging into a few-shot learning setting, The reviewers all agreed the work was sound; discussion after the author response focused on the theoretical justifications, degree of novelty and potential impact, and the empirical support. 

The primary concerns were that the work was slightly too incremental to obviously merit publication at this stage: though the empirical results were sound, they mostly follow the existing observation that SWA tends to be beneficial for generalization in other settings; apparently in few-shot learning as well. The positives would be that this is simple enough that it could become a general "best practice" in few-shot learning baselines, and as such communicating this is important.

The other discussion focused around the theoretical justification relating SWA to low-rank solutions. While empirically it does seem that the solutions found by SWA lead to low-rank representations, this is not really adequately explored, and it's not clear enough why this should be expected to happen. I think if this relationship between SWA and low-rank representations were more clearly explored then the paper would be a strong accept.

As it stands, it is quite borderline. Based on the scores (5,5,6), the recommendation is to reject, but it certainly could be included as well, as it has solid execution and is a clear topical fit for ICLR.
Decision: Reject