Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Update: I appreciate the response to address the major concerns. The proposed approach doesn't follow the episodic training, so there exists a clear difference from advanced MAML approaches, which update the task-specific parameters in the episodic training. I still believe that more empirical justifications should be required to decide which approach performs better than the others. On a positive side, I completely agree that it is non-trivial to develop a localized meta-learning framework from a theoretical perspective. So, I increased my score by one point. 

**Summary**
This work proposes a localized meta-learning framework that adaptively determines a hyperprior for a specific task, derived from a PAC-Bayesian analysis. 

**Detailed comments**
The main idea of this approach is to adjust the global hyperposterior distribution by the information of the input task. Conceptually, this approach is very similar to previous ones:
* Meta-learning with latent embedding optimization, ICLR’19.
* Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace, ICML’18.

Both propose data-dependent parameter generative processes, in which a subset of parameters is used to solve the input task. But, the proposed approach has not been compared to them.

The proposed method has been validated in very limited few-shot classification scenarios (only 5-way 50-shot tasks). It’s necessary to consider more standard experiments settings. I guess 1-shot / 5-shot on miniImagenet and tieredImagenet tasks should be included in experiments. Otherwise, it’s very difficult to see that this approach works well in general settings. 

For few-shot classification tasks, there exist many attempts to improve the performance. But, none of these approaches has not been compared to the proposed method. More empirical justification is needed.
review_title: This paper naturally extends PAC-Bayesian analysis to construct a task-adaptive hyperprior, but its empirical justification is not strong.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents an algorithm for offline meta-learning, where tasks are drawn from a distribution and presented to a learner sequentially, the objective being to use accumulated knowledge in order to facilitate the learning of new tasks. The algorithm is motivated from the PAC-Bayes theory of generalization, extended in recent years to the meta-learning setup, and provides a new localized approach where the prior used for a novel task is allowed to depend on the data from the new task in addition to all previous tasks. They make use of a previously proposed local learning method (LCC) that leads to extra flexibility and to a tightening of the meta-learning bounds, and provide an algorithm based on these bounds from which a learning algorithm is derived based on minimization of the upper bound. Empirical results are provided demonstrating the efficacy of the method and comparing to other recent approaches.  


Strong points The problems of meta-learning is of significant current interest and importance, and the theoretical formulation within the widely used and proven PAC-Bayes framework is well founded. Moreover, the authors go beyond current bounds by introducing a localized approach to data-dependent prior learning, deriving an algorithm directly from theoretical bounds, and demonstrating its utility.  

Weak points The paper is difficult to follow making it hard to assess its true novelty. For example, the notion of 'localized learning' is key to the work, but is never defined clearly. Given its important it warrants a clear definition. For instance, in the caption of image 1 they state ''Instead of using global
meta-knowledge for all tasks, we tailor the meta-knowledge for various specific task." This is not very helpful. If I understand correctly, the main issue is using the LCC approach for determining a data-dependent prior for the new task. Moreover, given the most work on PAC-Bayes theory specifically requires that the prior be data-independent, and that task-dependent priors have only very recently been introduced and analyzed (e.g., Rivasplata et al., PAC-Bayes analysis beyond the usual bounds, NeurIPS 2020 ), this is a major issue that requires clear elaboration.

Specific issues

The definition of coordinate coding and the latent manifold require some further elaboration, perhaps through an example. How do local anchor points relate to the standard notion of a basis? 

The authors assume that both the prior and posterior (lines 128-129) are spherical Gaussians with the same variance, i.e., they do not allow the variance to adapt through learning. Isn't this a significant limitation? The same holds for the distribution of v in line 160. Some discussion of this is called for. 

The authors present a bounds based on Catoni's work claiming, correctly, that it allows for faster convergence at a rate of 1/m rather than 1/\sqrt{m}. However, they refrain from noting that the empirical error is multiplied by a constant that is larger than 1, as opposed to previous bounds. While this may not be an issue for deep networks that have very low empirical error, this should be clearly stated. 

The form of the task-complexity term in eq. (3) and (9) requires further explanation and elaboration given its pivotal role in the work.

In Lemma 1 and Theorem 2, it is not clear whether you used the  Latent Manifold assumption (Def 3) , and where does it come into play in the bound and in the analysis?

The optimization of LCC is not explained sufficiently.  Since it is the main part of the innovation, it should be explained more than is done in Appendix B.  Eq. 12 uses L2 distance on the data points, how effective this is in real data sets?

Given the multiple definitions required for the algorithm, I would urge the authors to include all necessary definitions and equations in Appendix F that contains the pseudo-code. This will greatly facilitate understanding the exact form of the algorithm. 

Minor points
Line 49 - missing Fig number, Definition 3 - change order to  d_M:=|C|, Line 95 - delete b, Line 96 - any point in M (not in R^d). 

Concerns 
The overall idea of the paper is interesting, and, according to my understanding novel and promising. However, I cannot be certain that I have fully understood the paper, given the concerns raised above about lack of clarity. I believe that a clear re-writing of the paper, highlighting the novel contributions of the work and putting it in context (e.g., re new results by Rivasplata mentioned above) is called for. I believe that such a rewriting, assuming the correctness of the proofs in the appendix that I have not checked, will greatly enhance the paper. 

review_title: The overall idea of the paper is interesting, and, according to my understanding novel and promising. However, the lack of clarity and comparative analysis, renders my assement somewhat uncertain. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: ----------Updates after rebuttal------------

The author did not provide a revised version and additional experiments to address my questions. 

I keep my original score.

------------------------------------------


Summary: 

This paper proposed a new localized prior PAC-Bayes meta-learning. Specifically, they adopted local coordinate coding (LCC) to replace the previous global hyper-posterior predictive distribution.  Empirical results on few-shot tasks verified its practical benefits.

------------------------------------------------------

Overall review 

Pros:

[1] This problem is well-motivated. I think such a direction is meaningful. 

[2] The high level of proof is technically sound.

Cons:

[1] The paper organization indeed requires a better refinement. I can follow the main proof but too many non-important details make the paper rather difficult to follow. 

[2] (main concern) The theoretical and practical benefits are not significant. It seems like a plug-in approach for LCC.

[3] Some technical details need better justifications and discussions.

Based on these, I recommend a weak rejection but encourage a major revision for resubmission.

-----------------------------------------------------------
Detailed explanations

[1] Paper organization

The main contribution of this paper is Theorem 2 (and an improved Theorem 1). I think more intuitions/insights/discussions about these are expected. In contrast, the definition of Lipschitz and Smooth and Lemma 1,2 are not critical in understanding your main idea. 

This is particularly important in Sec 3, where the main contributions occur. But it is quite difficult to follow the insights in this part. And this also influences the significance of the paper.

[2] The significance of the results

Theoretical perspective:

[a] The author claimed they reach a tighter bound with $\mathcal{O}(\frac{1}{m})$ (Line 259). I respectfully disagree with this claim. In fact, the author chooses a specific function (smooth property) for deriving a tighter bound. But in the deep network due to the non-linear activation function, this condition can hardly be achieved in practice. Therefore the reason for improving the performance is not the proposed good theory.

[b] I cannot understand why using LCC in meta-Pac-Bayes.  Why not using other clustering-based approaches to set anchor points such as the core-set approach? If there is a **particular** benefit to use LCC rather than other localized approaches, the explanations are highly expected. 

Since the whole proof idea is quite similar to Amit (2018) and Pentina (2014) with two-level Pac-Bayes theorems except the LCC part. From the current version, it seems a simple LCC plug-in approach, which hurts its significance.  

Practical Perspective:

[c] The empirical results indicate LCC based PAC-Bayes heavily depend on the number of anchor numbers. i.e if we do not choose the best anchor number, the results can be even worse than the simple baseline ML_AM. 

[d] The computational complexity of LML is not reported. I think it can be much higher than ML_AM/ML_PL, which restricts its practical utility.

[e] The analysis of the result can be much better improved. For example, the author claimed better local information extraction. I would like to see clear evidence/analysis  (e.g. TSNE, visualization of anchor points, the relation of different tasks) rather than prediction accuracy curves. This also enhances the significance of LCC.


[3] Other minor details

[a] The loss in PAC-Bayes in $[0,1]$, the cross-entropy is surely not.

[b] The Catoni bound (eq 25) the $\pi$ must not depend on $x_1,\dots x_n$. This may contradict your data-dependent bound?

[c] Lemma 1 and Lemma 2 are straightforward. I do not think it does not provide strong practical insights into the proposed approach.  


--------------------------------------------

Suggestions 

I suggest a major revision on Sec 3, theoretical discussions, and empirical analysis (not only accuracy) on the benefits of LCC.

review_title: Review #1

final_score: rating: 5: Marginally below acceptance threshold
review_content: - Summary and Contributions
    - In this paper, the authors proposed a localized version of PAC-Bayes bound for meta-learning. This paper is a direct extension of (Amit and Meir 2018) by customizing the prior of each task with Local Coordinate Coding. 
		
- Strengths
    - The main text is easy to follow, with major theorems and intuitions well illustrated. 
    - The idea of localized prior for specific tasks is supported and inspired from the PAC-Bayes theory.
		
- Weaknesses and questions:
   - The theoretical novelty of this paper is limited - it is a combination of the theoretical analyses in (Amit and Meir 2018) and those in the work of LCC. 
   - Why do the hyperprior in Line 130 and the prior sampled from the hyper-posterior in Line 128 share the same variance? It does not make any sense. Actually, the two variances are different in the work of (Amit and Meir 2018).
   - Following the PAC-Bayesian theory and Line 154-156, the prior must be independent from the samples but could be selected from the distribution. However, in this paper, the prior w^P_i is determined from the samples S_i via LCC. I expect the theoretical analysis is not valid. 
   - From Equation (10), I do not expect the task complexity term to be reduced necessarily. Could the authors elaborate any guarantee of this algorithm to learn an effective v^Q so that this term is reduced?
   - What are the implications of c1 and c2?
   - In fact, there are many gradient-based meta-learning algorithms working on adapting the global meta-knowledge to specific tasks, including [1] and [2]. The authors did not compare any of them. Moreover, I am not convinced by the results in E.4. Figure 5. I am curious about LML's performance on standard mini-ImageNet and Omniglot datasets. 
    - This algorithm lacks flexibility in handling regression problems. Could the authors provide an explanation and discussion on this?
    - Could you illustrate the anchor points u empirically?
		
[1] Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim. Toward multimodal model-agnostic meta-learning. arXiv preprint arXiv:1812.07172, 2018.
[2] Yao, H., Wei, Y., Huang, J., & Li, Z. (2019, May). Hierarchically Structured Meta-learning. In International Conference on Machine Learning (pp. 7045-7054).
review_title: Limited theoretical novelty with some technical flaws

Meta-review: The paper presents a PAC-Bayesian approach for meta-learning that utilizes information of the task distribution in the prior. The presented localized approach allows the authors to derive an algorithm directly from the bound - this is a worthwhile contribution. Nevertheless there are several concerns that were raised by the reviewers and in its current form the work is not ready to appear in ICLR.  


Decision: Reject