Localized Meta-Learning: A PAC-Bayes Analysis for Meta-Learning Beyond Global Prior

Abstract: Meta-learning methods learn the meta-knowledge among various training tasks and aim to promote the learning of new tasks under the task similarity assumption. Such meta-knowledge is often represented as a fixed distribution; this, however, may be too restrictive to capture various specific task information because the discriminative patterns in the data may change dramatically across tasks. In this work, we aim to equip the meta learner with the ability to model and produce task-specific meta-knowledge and, accordingly, present a localized meta-learning framework based on the PAC-Bayes theory. In particular, we propose a Local Coordinate Coding (LCC) based prior predictor that allows the meta learner to generate local meta-knowledge for specific tasks adaptively. We further develop a practical algorithm with deep neural network based on the bound. Empirical results on real-world datasets demonstrate the efficacy of the proposed method.

Reviews and author responses:

Update: I appreciate the response to address the major concerns. The proposed approach doesn't follow the episodic training, so there exists a clear difference from advanced MAML approaches, which update the task-specific parameters in the episodic training. I still believe that more empirical justifications should be required to decide which approach performs better than the others. On a positive side, I completely agree that it is non-trivial to develop a localized meta-learning framework from a theoretical perspective. So, I increased my score by one point. 

**Summary**
This work proposes a localized meta-learning framework that adaptively determines a hyperprior for a specific task, derived from a PAC-Bayesian analysis. 

**Detailed comments**
The main idea of this approach is to adjust the global hyperposterior distribution by the information of the input task. Conceptually, this approach is very similar to previous ones:
* Meta-learning with latent embedding optimization, ICLR’19.
* Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace, ICML’18.

Both propose data-dependent parameter generative processes, in which a subset of parameters is used to solve the input task. But, the proposed approach has not been compared to them.

The proposed method has been validated in very limited few-shot classification scenarios (only 5-way 50-shot tasks). It’s necessary to consider more standard experiments settings. I guess 1-shot / 5-shot on miniImagenet and tieredImagenet tasks should be included in experiments. Otherwise, it’s very difficult to see that this approach works well in general settings. 

For few-shot classification tasks, there exist many attempts to improve the performance. But, none of these approaches has not been compared to the proposed method. More empirical justification is needed.

----------Updates after rebuttal------------

The author did not provide a revised version and additional experiments to address my questions. 

I keep my original score.

------------------------------------------


Summary: 

This paper proposed a new localized prior PAC-Bayes meta-learning. Specifically, they adopted local coordinate coding (LCC) to replace the previous global hyper-posterior predictive distribution.  Empirical results on few-shot tasks verified its practical benefits.

------------------------------------------------------

Overall review 

Pros:

[1] This problem is well-motivated. I think such a direction is meaningful. 

[2] The high level of proof is technically sound.

Cons:

[1] The paper organization indeed requires a better refinement. I can follow the main proof but too many non-important details make the paper rather difficult to follow. 

[2] (main concern) The theoretical and practical benefits are not significant. It seems like a plug-in approach for LCC.

[3] Some technical details need better justifications and discussions.

Based on these, I recommend a weak rejection but encourage a major revision for resubmission.

-----------------------------------------------------------
Detailed explanations

[1] Paper organization

The main contribution of this paper is Theorem 2 (and an improved Theorem 1). I think more intuitions/insights/discussions about these are expected. In contrast, the definition of Lipschitz and Smooth and Lemma 1,2 are not critical in understanding your main idea. 

This is particularly important in Sec 3, where the main contributions occur. But it is quite difficult to follow the insights in this part. And this also influences the significance of the paper.

[2] The significance of the results

Theoretical perspective:

[a] The author claimed they reach a tighter bound with $\mathcal{O}(\frac{1}{m})$ (Line 259). I respectfully disagree with this claim. In fact, the author chooses a specific function (smooth property) for deriving a tighter bound. But in the deep network due to the non-linear activation function, this condition can hardly be achieved in practice. Therefore the reason for improving the performance is not the proposed good theory.

[b] I cannot understand why using LCC in meta-Pac-Bayes.  Why not using other clustering-based approaches to set anchor points such as the core-set approach? If there is a **particular** benefit to use LCC rather than other localized approaches, the explanations are highly expected. 

Since the whole proof idea is quite similar to Amit (2018) and Pentina (2014) with two-level Pac-Bayes theorems except the LCC part. From the current version, it seems a simple LCC plug-in approach, which hurts its significance.  

Practical Perspective:

[c] The empirical results indicate LCC based PAC-Bayes heavily depend on the number of anchor numbers. i.e if we do not choose the best anchor number, the results can be even worse than the simple baseline ML_AM. 

[d] The computational complexity of LML is not reported. I think it can be much higher than ML_AM/ML_PL, which restricts its practical utility.

[e] The analysis of the result can be much better improved. For example, the author claimed better local information extraction. I would like to see clear evidence/analysis  (e.g. TSNE, visualization of anchor points, the relation of different tasks) rather than prediction accuracy curves. This also enhances the significance of LCC.


[3] Other minor details

[a] The loss in PAC-Bayes in $[0,1]$, the cross-entropy is surely not.

[b] The Catoni bound (eq 25) the $\pi$ must not depend on $x_1,\dots x_n$. This may contradict your data-dependent bound?

[c] Lemma 1 and Lemma 2 are straightforward. I do not think it does not provide strong practical insights into the proposed approach.  


--------------------------------------------

Suggestions 

I suggest a major revision on Sec 3, theoretical discussions, and empirical analysis (not only accuracy) on the benefits of LCC.


Thank you for your thoughtful review. We answer your questions below.

Q1: The theoretical novelty of this paper is limited.
A1: Please refer to our response A2 to R1

Q2: hyperprior and hyperposterior share the same variance.
A2; Please refer to our response to A3 to R4

Q3:  the prior w^P_i is determined from the samples S_i via LCC
A3: Please refer to our response A5 to R1

Q4: From Equation (10), I do not expect the task complexity term to be reduced necessarily.
A4: Instead of using a fixed distribution, we formulate meta-knowledge as a flexible mapping function from task data distribution to specific prior. As claimed in line76, we provide a means to tighten the PAC-Bayes meta-learning bound. Empirical results in Fig 4(b) validate the efficacy of the proposed method. Specifically, to make the proposed prior predictor close to the posterior, we summarize the required properties in line166-176.

Q5: What are the implications of c1 and c2?
A5: c1, c2 can be treated as hyperparameters that balance the trade-off between empirical error and regularization terms.

Q6: More baseline and dataset from meta-learning and few-shot learning
A6; Please refer to our response to R3

Q7: This algorithm lacks flexibility in handling regression problems.
A7: We respectively disagree with it. We think the proposed LML could be easily extended to regression problems. AS we discussed it in line 166-176, the key step of designing a good prior predictor for classification problems should satisfy these three properties.  For the regression problem, the prior predictor should be regress task agnostic since it will be used in the new regression task.

We greatly appreciate your comments and thoughtful suggestions. We respond to each comment as follows.

Q1:Why using LCC.
 A1: We have explained the properties required for constructing prior predictor in line 166-176. Here we propose an LCC-based prior predictor as an example. Because it provides a means to approximate the high-dimensional non-linear function with a global linear function with theoretical guarantees.

Q2: The framework seems a simple LCC plug-in approach
A2: We follow the two-level proof framework proposed in Amit (2018) and Pentina (2014), but we respectively disagree that it is a simple LCC plug-in approach. In the original setting, meta-knowledge is formulated as a hyperposterior distribution that generates the prior for each task. In LML, we formulate meta-knowledge as a mapping function from task data distribution to the prior, which makes it not trivial to address. Thus, we propose Lemma 1, 2, and LCC based prior predictor, which is novel in itself.

Q3: The choice of anchor numbers. Computation complexity
A3: As we claimed in line220-224, if we set anchor numbers to 1, it degenerates to the regular meta-learning framework. If we set it to a quite large number, it will suffer from the overfitting issue. We respectively disagree this flexibility is a flaw of LML.  As we described in line 240-243, we utilize an auto-encoder to extract the semantic information of data and then construct the LCC scheme based on the embeddings.  The extra computation cost scales linearly to anchor numbers and embedding size, which is relatively small compared to the base model parameter.

Q4: The loss in PAC-Bayes in [0,1], the cross-entropy is surely not.
A4: We discussed it in line 476-478.

Q5: The Catoni bound (eq 25) the π must not depend on x1,…xn. This may contradict your data-dependent bound?
A5: We respectively disagree with it. We discussed it in line191, we choose prior upon the task data distribution. Since it is unknown, we approximate it by its empirical counterpart.  A similar idea has been used in “PAC-Bayes  bounds with data-dependent priors” and “Data-dependent PAC-Bayes priors via differential privacy”

We thank the reviewer for the valuable feedback and helpful comments. We will reorganize the structure to make it more clear in the next version. Here we answer your questions below.

Q1: “The notion of 'localized learning' is key to the work…require clear elaboration”
A1:  Your understanding is correct. We present the formal definition in line 157-165 and the localized meta-learning is implemented with an LCC-based prior predictor. We discuss the data-independent and data-dependent priors in line 140-155. Thanks for point out the paper “PAC-Bayes analysis beyond the usual bounds”. We will add and discuss it in our next version.

Q2:” How do local anchor points relate to the standard notion of a basis”
A2: Thanks for your suggestion. Due to space limitations, we have removed the example for LCC in the current version. We will put it in the appendix in our next version. Here the anchor point and the basis have the same meaning.

Q3: “Both the prior and are spherical Gaussians with the same variance, i.e., they do not allow the variance to adapt through learning. Isn't this a significant limitation?
A3: In the current version, we follow the setting in “PAC-Bayes bounds with data-dependent priors.” and formulate the prior predictor as a mapping function from task data distribution to the mean of the base model parameter. We only consider the mean for ease of presentation. It should be noted that our framework can be easily extended to the setting considering both the mean and the variance.

Q4: Catoni's bound
A4: As we claimed in line 121-122, the proof technique is agnostic to the specific single-task bound. To make a fair comparison, we adopt the same Catoni’s bound to analyze the regular ML and the proposed LML framework.

Q5: Further explanation about eq. (3) and (9)
A5: Our explanation can be found in line 210-224

Q6: The use of Latent Manifold assumption
A6: Due to space limitation, we put the further analysis about lemma 1 in the appendix in line 429-431, which uses latent manifold assumption to derive the upper bound in eq. (14)

Q7: Optimization of LCC
A7: As described in line 241-243, we use an auto-encoder to extract the semantic information of image data and then construct the LCC scheme based on the embeddings. The LCC is optimized with an alternative update between C and \gamma. More details for LCC training can be found in line 500-506.

We thank the reviewer for raising your concern in experiments. We agree that some existing meta-learning works follow the similar concept, however, they are not derived from the generalization bound. We discuss these in related work in line395-406. Thanks for pointing out these two works. We will include them in the related work section. We want to emphasize that we do not follow the episodic training to address few-shot learning but to explore the PAC-Bayes meta-learning framework [Amit et al. 2018]. As we discussed in line225, to make a fair comparison with PAC-Bayes baselines, we follow the same joint optimization method as Amit et al. 2018 to ensure that the benefit of LML is not from using an improved optimization method. Thus, we also follow the similar meta-learning environment setting as Amit et al. 2018. That is, the meta-learner observes a limited number of tasks (from 1 to 11) and each task has sufficient samples. In this scarce task setting, conventional meta-learning methods following the episodic training paradigm will suffer from the severe task-overfitting issue. We can observe LML significantly outperforms MAML and MatchingNet in Fig 5 and we explain it in line531-538. Thus, we do not compare any improved meta-learning method that follows the episodic training paradigm. 

The paper presents an algorithm for offline meta-learning, where tasks are drawn from a distribution and presented to a learner sequentially, the objective being to use accumulated knowledge in order to facilitate the learning of new tasks. The algorithm is motivated from the PAC-Bayes theory of generalization, extended in recent years to the meta-learning setup, and provides a new localized approach where the prior used for a novel task is allowed to depend on the data from the new task in addition to all previous tasks. They make use of a previously proposed local learning method (LCC) that leads to extra flexibility and to a tightening of the meta-learning bounds, and provide an algorithm based on these bounds from which a learning algorithm is derived based on minimization of the upper bound. Empirical results are provided demonstrating the efficacy of the method and comparing to other recent approaches.  


Strong points The problems of meta-learning is of significant current interest and importance, and the theoretical formulation within the widely used and proven PAC-Bayes framework is well founded. Moreover, the authors go beyond current bounds by introducing a localized approach to data-dependent prior learning, deriving an algorithm directly from theoretical bounds, and demonstrating its utility.  

Weak points The paper is difficult to follow making it hard to assess its true novelty. For example, the notion of 'localized learning' is key to the work, but is never defined clearly. Given its important it warrants a clear definition. For instance, in the caption of image 1 they state ''Instead of using global
meta-knowledge for all tasks, we tailor the meta-knowledge for various specific task." This is not very helpful. If I understand correctly, the main issue is using the LCC approach for determining a data-dependent prior for the new task. Moreover, given the most work on PAC-Bayes theory specifically requires that the prior be data-independent, and that task-dependent priors have only very recently been introduced and analyzed (e.g., Rivasplata et al., PAC-Bayes analysis beyond the usual bounds, NeurIPS 2020 ), this is a major issue that requires clear elaboration.

Specific issues

The definition of coordinate coding and the latent manifold require some further elaboration, perhaps through an example. How do local anchor points relate to the standard notion of a basis? 

The authors assume that both the prior and posterior (lines 128-129) are spherical Gaussians with the same variance, i.e., they do not allow the variance to adapt through learning. Isn't this a significant limitation? The same holds for the distribution of v in line 160. Some discussion of this is called for. 

The authors present a bounds based on Catoni's work claiming, correctly, that it allows for faster convergence at a rate of 1/m rather than 1/\sqrt{m}. However, they refrain from noting that the empirical error is multiplied by a constant that is larger than 1, as opposed to previous bounds. While this may not be an issue for deep networks that have very low empirical error, this should be clearly stated. 

The form of the task-complexity term in eq. (3) and (9) requires further explanation and elaboration given its pivotal role in the work.

In Lemma 1 and Theorem 2, it is not clear whether you used the  Latent Manifold assumption (Def 3) , and where does it come into play in the bound and in the analysis?

The optimization of LCC is not explained sufficiently.  Since it is the main part of the innovation, it should be explained more than is done in Appendix B.  Eq. 12 uses L2 distance on the data points, how effective this is in real data sets?

Given the multiple definitions required for the algorithm, I would urge the authors to include all necessary definitions and equations in Appendix F that contains the pseudo-code. This will greatly facilitate understanding the exact form of the algorithm. 

Minor points
Line 49 - missing Fig number, Definition 3 - change order to  d_M:=|C|, Line 95 - delete b, Line 96 - any point in M (not in R^d). 

Concerns 
The overall idea of the paper is interesting, and, according to my understanding novel and promising. However, I cannot be certain that I have fully understood the paper, given the concerns raised above about lack of clarity. I believe that a clear re-writing of the paper, highlighting the novel contributions of the work and putting it in context (e.g., re new results by Rivasplata mentioned above) is called for. I believe that such a rewriting, assuming the correctness of the proofs in the appendix that I have not checked, will greatly enhance the paper. 


- Summary and Contributions
    - In this paper, the authors proposed a localized version of PAC-Bayes bound for meta-learning. This paper is a direct extension of (Amit and Meir 2018) by customizing the prior of each task with Local Coordinate Coding. 
		
- Strengths
    - The main text is easy to follow, with major theorems and intuitions well illustrated. 
    - The idea of localized prior for specific tasks is supported and inspired from the PAC-Bayes theory.
		
- Weaknesses and questions:
   - The theoretical novelty of this paper is limited - it is a combination of the theoretical analyses in (Amit and Meir 2018) and those in the work of LCC. 
   - Why do the hyperprior in Line 130 and the prior sampled from the hyper-posterior in Line 128 share the same variance? It does not make any sense. Actually, the two variances are different in the work of (Amit and Meir 2018).
   - Following the PAC-Bayesian theory and Line 154-156, the prior must be independent from the samples but could be selected from the distribution. However, in this paper, the prior w^P_i is determined from the samples S_i via LCC. I expect the theoretical analysis is not valid. 
   - From Equation (10), I do not expect the task complexity term to be reduced necessarily. Could the authors elaborate any guarantee of this algorithm to learn an effective v^Q so that this term is reduced?
   - What are the implications of c1 and c2?
   - In fact, there are many gradient-based meta-learning algorithms working on adapting the global meta-knowledge to specific tasks, including [1] and [2]. The authors did not compare any of them. Moreover, I am not convinced by the results in E.4. Figure 5. I am curious about LML's performance on standard mini-ImageNet and Omniglot datasets. 
    - This algorithm lacks flexibility in handling regression problems. Could the authors provide an explanation and discussion on this?
    - Could you illustrate the anchor points u empirically?
		
[1] Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim. Toward multimodal model-agnostic meta-learning. arXiv preprint arXiv:1812.07172, 2018.
[2] Yao, H., Wei, Y., Huang, J., & Li, Z. (2019, May). Hierarchically Structured Meta-learning. In International Conference on Machine Learning (pp. 7045-7054).

Meta-review: The paper presents a PAC-Bayesian approach for meta-learning that utilizes information of the task distribution in the prior. The presented localized approach allows the authors to derive an algorithm directly from the bound - this is a worthwhile contribution. Nevertheless there are several concerns that were raised by the reviewers and in its current form the work is not ready to appear in ICLR.  



Decision: withdrawn-rejected-submissions