Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a novel and effective meta-learning algorithm using the so-called Eigen-Reptile to update the meta-parameters and address the gradient noises. To further improve the accuracy of the main direction of Eigen-Reptile, the authors employ self-paced learning algorithm to construct several prior models determining which samples should be abandoned. The experiments demonstrate strong robustness against label noises.

Strength:
- The proposed algorithm is technically sound. The authors provide the solution to address the high cost of computing eigenvalue and eigenvector in the ER process, and also offer the theorem to demonstrate that the eigenvectors will not be affected by gradient noises. 
- The experiments are solid to demonstrate the usefulness of the Eigen-Reptile algorithm. Applying ER in regression, few-shot classification tasks achieve superior performance and robustness compared to the selected baseline models.

Weakness:
- Missing analyses on the training and testing overhead, e. .g, runtime, running memory.
- Missing comparisons to more recent meta-learning and few-shot learning approaches. 
- ER is built upon a CONV-4 backbone. Is it applicable to more complicated architectures, e. g. ResNet-12 considering both the performance and overhead.
review_title: An novel and robust meta-learning algorithm

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper is concerned about the update of meta-parameters in gradient-based meta-learning approaches. Developed upon the recent method called Reptile (Nichol et al., 2018), this work aims to make it better deal with the sampling and label noises in few-short learning. The motivation is that Reptile updates meta-parameters towards the last task-specific parameters, but this could lead to biased updating. Instead, this work proposes to update the meta-parameters by the main direction of historical task-specific parameters to reduce the variation on gradients caused by the sampling and label noises. Theoretical analysis is conducted to show that the main direction of historical task-specific parameters is just the leading eigenvector of the covariance matrix computed upon the task-specific parameters obtained in the $n$ inner loop steps. This method is called Eigen-Reptile in this paper accordingly. 

Further, this work proposes Introspective Self-paced Learning (ISPL) to improve the precision of the estimate of the main direction above. It creates multiple prior models by randomly sampling the training data and discards the samples with high loss values. Theoretical analysis is conducted to justify the proposed ISPL method. 

Experimental study is conducted on a synthetic dataset and the Mini-ImageNet benchmark in few-shot classification tasks. The result shows that the proposed method can achieve better performance than Reptile and other state-of-the-art methods. 

Overall, the issue researched in this paper has its significance, and the idea is sound. Also, experimental study demonstrates the effectiveness of the proposed method. 

The main weaknesses of this work lie at its theoretical part, as detailed as follows.

1. Essentially, this work can be regarded as a kind of denoise method to better update the meta-parameters. Different from Reptile, this work proposes to update the meta-parameters by the "main direction" of task-specific parameters. Why is the "main direction" the best choice? The simplest choice could be the "mean" of the $n$ task-specific parameters. How is this option compared with the main direction? Please clarify. 

2. The theoretical analysis in Section 4.1 is not novel. It is largely a derivation of principal component analysis which can be founded in textbooks. The only difference is that a linear constraint $\bar{V}e>0$ is imposed in Eq.(3). However, if understood correctly, the derivation seems to be flawed. This linear constraint does not have any effect to the optimal solution of Eq.(3), which is still proved as the leading eigenvector of $S$. Eq.(3) is a linear constrained quadratic programming. The leading eigenvector of $S$ does not necessarily satisfy the linear constraint related to an arbitrary $\bar{V}$. This may be related to some issue in the KKT condition listed in Eq.(5). Please double check or clarify it. 

3. The trick in Eq.(6) is not new in machine learning and therefore cannot be regarded as a technical contribution. This trick has long been used to compute the eigensystems of the covariance matrix of "thin" or "fat" data matrices, for example, in the seminal work [R1]. 

4. The proof in Appendix B is largely routine. It mainly shows the properties of a covariance matrix when data are corrupted by white noise (that is, following a Gaussian distribution).

5. The proposed ISPL is interesting. However, the theoretical proof for Theorem 2 in the Appendix needs to be made clear. Particularly, the statements between Eq(29) and (30) are vague. Also, ISPL seems to be more related to ensemble learning. This paper may utilize existing ensemble learning theories to better justify this idea. 

In short, this work is well motivated and the overall idea is sound. Experimental study supports the effectiveness of the proposed methods. Particularly, this paper makes efforts to provide theoretical justification for each of the methods, which is appreciated. However, these theoretical justifications currently are not strong or novel enough. Some of them need to be double checked and some need to be further clarified.  

[R1] M. Turk; A. Pentland (1991). "Face recognition using eigenfaces" (PDF). Proc. IEEE Conference on Computer Vision and Pattern Recognition. pp. 586–591.

--- Thank the authors for the detailed response. After reading the response and the comments of peer reviewers, it is still felt that this work needs to better clarify some key issues and strengthen both theoretical and experimental study. In light of these, the rating is maintained as follows. 
review_title: An interesting paper but has weakness on the theoretical part

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a reptile-based meta-learning algorithm called Eigen-Reptile for few shot learning with sampling and label nosing. When Eigen-Reptile updates meta-parameters, it leverages not only the gradient direction of different task, but also the direction of eigenvector related to parameters matrix. Besides, authors propose Introspective Self-paced Learning (ISPL) for label noise problem. N-way-K-shot experiments on Mini-Imagenet demonstrate the effectiveness of Eigen-Reptile and 5-way-1shot with symmetric label noise experiments on Mini-Imagenet show that ISPL could alleviate the noising problem in some degree.

Strength:
1. The idea of Eigen-Reptile to alleviate gradient noise by eigenvector of parameters-related matrix is interesting and authors prove the effectiveness of the idea in theory.
2. A clever, avoiding high time complexity way to obtain the eigenvector of parameters-related matrix
3. By idea of self-paced learning with prior model to solve noisy few shot problem is reasonable and ingenious with theoretical proof.

Weakness:
1. The writing is confusing and not clear enough. For example, Para. 4.1, line 6. What is the specific meaning of “main direction of n points”, and what is suitable mathematical expression of “the unit vector e” ?
2. In Mini-Imagenet N-way K-shot experiments, authors didn’t show specific numbers of filter of the most important comparison object,  Reptile, and the final experiments results are not particularly outstanding compared with recent papers, like DPGN[1], SIB[2].
3. For label noise experiments, it is hard to say the ISPL is indeed effective as results showed in line 1,2,3 of Table 2 in paper.


[1] Yang, Ling, et al. "DPGN: Distribution Propagation Graph Network for Few-shot Learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020
[2] Hu, Shell Xu, et al. "Empirical Bayes Transductive Meta-Learning with Synthetic Gradients." ICLR (2020)..
review_title: This paper presents a reptile-based meta-learning algorithm called Eigen-Reptile for few shot learning with sampling and label nosing. When Eigen-Reptile updates meta-parameters, it leverages not only the gradient direction of different task, but also the direction of eigenvector related to parameters matrix. Besides, authors propose Introspective Self-paced Learning (ISPL) for label noise problem. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presents a method about robust meta-learning by analyzing eigenvalues and eigenvectors. The ultimate goal is to learn the new task and denoise the gradients when updating the model parameters for the specific task.  There are two aspects in the proposed approach to alleviate gradient noise: 1). Parameters updates with eigen vectors. 2). introspective self-paced learning to correct the loss of noisy samples. 


Strengths:
- The problem of meta-learning with noisy labels is introduced in this work with some experiments to show the robustness of the proposed approach.
- The analysis with various noise percentages shows that meta-learning suffers from noisy labels.
 
Weaknesses:
- The manuscript is very hard to follow especially some important information is located in Appendix. For instance, in the main paper it is not very clear how the algorithm works and implemented, this important information is provided in Appendix (Algorithm 1 and 2).
- It is not very clear how the inner-update is performed because U(.) can be any meta-learning algorithm but the work only focuses on Reptile. The inner-loop should be explicitly defined. 
- This work has no analysis about the limit of noise that is acceptable to the algorithm (especially related to the eigenvalues). 
- The assumption of Theorem 1 is strong that the gradient contains Gaussian noise from the noisy labels. Is there any empirical evidence/previous work providing the fact or the proof?
- The experiment settings are very artificial and the problem is not well-motivated. How is it possible to have a lot of noisy labels when only a few data is provided? Is there any valid reason or practical example to justify the setup in this work?
- The literature review of this work is very poor:
1. Meta-learning with differentiable closed-form solvers is proposed by Bertinetto et al. [1]. Is there any reason to cite the technical report (reimplementation) by Devos et al.?
2. Gradient noise in meta-learning has been investigated by Simon et al. [2] but it is not discussed and compared in the manuscript.
3. A recent method called Meta-Curvature [3] which is a closely correlated topic to the proposed method is not discussed and compared.

- The experiments are not convincing to show the superiority of the proposed method:
1. The dataset is limited to one dataset for classification tasks. This paper only provides the mini-ImageNet dataset.
2. A rigorous experiment is needed by varying the parameters (e.g. the number of shot, the number of inner-loop, and the number of way) to show the robustness are not shown. This is important because the setup is new but the experiments do not show various conditions.
3. There is no comparison with the existing methods for combating noisy labels. Some approaches [4, 5, 6] can be adopted to the loss function for inner-loop.


There are some questions that need to be addressed:
1. In Eq. 3, is it possible that \bar{V}.e is zero? Because the inner-loop can reach a stationary point.
2. Is ISPL applicable to any learning condition? What is the special thing of this loss function in meta-learning?


[1] L. Bertinetto, J. F. Henriques,  P. Torr, and A. Vedaldi, "Meta-learning with differentiable closed-form solvers," ICLR, 2018.

[2] C. Simon, P. Koniusz, R. Nock, M. Harandi, "On modulating the gradient for meta-learning," ECCV, 2020. 

[3] E. Park and J.B. Olivia, "Meta-curvature," Neurips, 2019.

[4] E. Arazo, D. Ortego, P. Albert, N. E. O'Connor, K. McGuiness,"Unsupervised Label Noise Modeling and Loss Correction," ICML, 2019.

[5] Y.Wang, X. Ma, Z. Chen, Y. Luo, J. Yi, J. Bailey, "Symmetric Cross Entropy for Robust Learning with Noisy Labels,"ICCV, 2019.

[6] G. Patrini, A. Rozza, A. K. Menon, R. Nock, L. Qu,"Making Deep Neural Networks Robust to Label Noise:
a Loss Correction Approach," CVPR, 2017.



--------------------------------------------------------------------------------------------------------------------------------------------------

I want to thank the authors for the revision and the rebuttal. Even though the proposed method is promising for meta-learning with noisy gradients, the setup is not backed up with strong arguments/facts. Moreover, I do not fully agree to the statements in the response. For instance, these arguments are not fully observable neither by experiments nor theories:

1. " Specifically, the meta-learner is prone to meta-overfit, as there are only a few available samples with sampling noise, even data is clean.". There is no proof that the meta-learner is prone to meta-overfit. The 'meta-overfit' term is not defined well. What I understand is that the learned meta-parameter is prone to overfitting in this context.
2. "Due to the small amounts of samples, FSL is more easily affected by data noise, especially considering that human annotators are likely to make mistakes as training meta-learner requires a large number of classes. Besides, we are not the first to propose noisy FSL, [10] propose hybrid attention-based prototypical networks for the noisy few-shot relation classification task, as human annotators are also likely to make mistakes in language tasks.". A common sense is that a few data is more managable and less vulnerable to mislabeling.
3. "Our noisy labels experiments aim to verify the robustness of Eigen-Reptile to noisy labels. Denoise is not the focus of our research. At the same time, ISPL is not a denoise algorithm in the traditional sense.". The statement is vague and not aligned to the introduction, abstract, and title of this work.

The theoretical part is mostly trivial as mentioned by R2, and the contribution (Algorithm) is hidden in Appendices. I acknowledge the author's response by increasing the score but I think the paper needs to be further revised.

I suggest to build strong background of this topic (meta-learning + noise/label noise) and relevant experiments to show the effectiveness of the proposed approach. Furthermore, some theoretical analysis about the limit to the noise can be verified for the future direction. For experiments with noisy gradients, I suggest to compare with the baseline on meta-learning + noise, e.g. the work by Simon et al. [2]. If the focus was noisy labels then it would be better to compare with other methods for noisy labels (not necessarily in the meta-learning setup).
review_title: The manuscript needs a major revision. The experiments and analysis are poor.

Meta-review: This paper was evaluated by four reviewers. After rebuttal, several concerns remained, e.g. Rev. 1 is interested in more thorough comparisons even if the model is claimed to be backbone-agnostic. Rev. 2 is concerned about re-print of some theories and authors' response that 'contribution is not in theoretical innovation'. Rev. 3 is overall not impressed with the clarity of the paper. Finally, Rev. 4 also remains unconvinced after rebuttal due to several somewhat loose explanations provided by authors.

At this point, AC agrees with reviewers that the paper requires more clear-cut theoretical contributions, ablations and improvements in writing clarity. While some reviewers might have been more inspired by the aspect of noisy labels, even ignoring this aspect, the overall consensus among all reviewers stands.
Decision: Reject