Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors proposed a novel meta-learning approach referred to as Meta-ticket, to find optimal sparse subnetworks for few-shot learning within randomly initialized NNs. 

They empirically validated that Meta-ticket successfully finds subnetworks that can learn specialized features for each given task. 

Due to this rapid task-wise adaptation ability, Meta-ticket achieves superior meta generalization compared to MAML-based methods, especially with large NNs. 

strengths_and_weaknesses: The authors proposed a new Meta-ticket for few-shot learning, inspired by Lottery Ticket Hypothesis.

The Meta-ticket finds optimal sparse subnetworks which tend to adapt to each task with a large change in the overall parameters. 

(+) The Meta-ticket is technically sound and seems to be supported by several experimental results. 

(+) For fair comparisons with others, the authors prepared the cross-domain experimental settings and baselines – MAML, ANIL, and BOIL. 

(-) The authors only showed the parts of adaptations such as general to specific and general to general.

(-) There is no ablation study on the layerwise threshold hyperparameter which seems to be sensitive to the initialization of weight scores.


questions: The Authors stated that Meta-ticket prefers rapid learning rather than feature reuse. On similar cross-domain meta-tasks, feature reuse might be preferred over rapid learning. 

Is there any case that feature reuse is better than rapid learning in the Meta-ticket (adaptations such as specific to general or specific to specific)? A plot like Fig.2 would be helpful for better analysis of other adaptations.

The effects of the initial sparsity in B.2 (Fig.1 and Table 6) were a good analysis. The layerwise threshold hyperparameter seems to be sensitive to the initialization of weight scores. The authors could prepare the sensitivity analysis on various kinds of initialization methods with their iteration-validation performances.


limitations: The authors showed the limitation of subnetworks; Meta-ticket tends to degrade with small NNs compared to MAML.

To the issue, this work has found the solution to the degradation caused by the small NNs, providing iterative randomization techniques.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a novel gradient-based meta-learning method, Meta-Ticket, as an alternative of MAML. For meta-objective, Meta-Ticket aims to optimize a sparse subnetwork hidden within a randomly initialized neural network (meta-mask, called meta-ticket), while MAML optimizes the initial parameters (meta-initialization). 

The experiments show that, Meta-ticket can achieve superior performance than MAML on cross-domain few-shot settings (over different datasets), although it tends to achieve worse accuracies than MAML in general setting.
strengths_and_weaknesses: Pros:
1. If focusing on cross-domain few-shot applications, the proposed Meta-Ticket achieves better meta-generalization ability rather than MAMLs.
2. Sufficient experiments over many datasets (domains) focusing on different cross-domain tasks and different types of NNs.

Cons:
1. Novelty is limited. The proposed Meta-Ticket algorithm is almost the same as ProsPr [1], which also optimizes a subnetwork (learns a mask $\mathbf{m}$) AND uses meta-gradients to take into account that we are going to train the weights after we apply mask to get subnetwork. 

However, there are two differences between this paper and [1]:
    (1) The x <- UpdateScores(s, L) in the paper replaces the "saliency scores computation" in [1] with the technique in [37].
    (2) The training steps after pruning, in this paper, is obtained from the few-shot validation data from multiple tasks. But in [1], the training steps after pruning is standard training from large data of single task. 

Despite the two differences, I still think the novelty of the proposed algorithm is not enough.

2. As pointed out in the paper (Section 6 and 4.1.1), the Meta-Ticket may tend to optimize larger network to achieve similar results as MAML, as it tries to find a hidden prune-mask within the fixed random parameters. This could be a limitation (i.e., memory problem, speed issue, etc) if training machines do not have enough resources. 
3. In experiments, missing a lot of SOTA gradient-based meta-learning methods. Especially, there are many task-specific gradient-based methods (e.g., LEO, MMAML) that is good for cross-domain few-shot learning. Maybe the authors should consider including them.

[1] Milad Alizadeh, Shyam A. Tailor, Luisa M Zintgraf, Joost van Amersfoort, Sebastian Far- quhar, Nicholas Donald Lane, and Yarin Gal. Prospect pruning: Finding trainable weights at initialization using meta-gradients. In International Conference on Learning Representations, 2022.

[37] Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What’s hidden in a randomly weighted neural network? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11893–11902, 2020.
questions: If ProsPr [1] is applied to a single few-shot task, can it achieve the same results as Meta-ticket (that is trained over many few-shot tasks) on testing few-shot tasks?
limitations: See above
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This works presents a few-shot learning framework named Meta-Ticket based on gradient-based meta learning. The meta learning is applied to select optimal sub-network from a randomly initialised network, so it’s easier to transfer to a novel domain, as similar to MAML. Meta-ticket has shown to achieve comparable, most of the time better performance than MAML, particularly in larger NN setting, and with more difficult datasets. The paper additionally provides interesting discussions on learning behaviour of few-shot adaptation in comparison with MAML, both in empirical and theoretical perspective. 
strengths_and_weaknesses: The paper is very easy to follow and supported with a clear and interesting motivation. In general, the authors present a very novel way of doing few-shot learning, combing some of different research areas of machine learning of Lottery ticket hypothesis, randomly initialised network and meta learning.  The discussion on rapid learning v.s. feature reuse is one hightlight of this work, to better understand the learning strategies of different few-shot learning methods. The authors also support the observation with theoretical argument.

Some additional comments which may further improve this work:

-- Search the optimal sub-network is a well-studied technique in transfer/multi-task learning strategy. I would suggest adding relevant related works and slightly discuss the difference of the proposed mask searching strategy: [1,2,3].

[1] Mallya et al. Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights, 2018.

[2] Hung et al. Compacting, Picking and Growing for Unforgetting Continual Learning 2019.

[3] Wortsman et al. Supermasks in Superposition 2020.

-- After searching the optimal subnetwork, I assume the rest of the network parameters are remained unused? If so, the baseline methods were trained with a larger parameter size? This actually leads to unfair comparison, as in the few-shot learning literature, it may leads to degraded performance due to overfitting. 

-- Is there any hyper-parameter controlling the size of the subnetwork? It would be interesting to see how different size of subnetwork can relate to the final few-shot learning performance.

 
questions: See the weaknesses.
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose Meta-ticket as a meta-learning approach that finds optimal sparse subnetworks for few-shot learning within the randomly initialized neural network. The paper can be seen as a MAML modification. 
strengths_and_weaknesses: Strengths 
1. The paper is well written and easy to follow.
2. The idea looks fresh and interesting.
3. The analysis of rapid learning and feature reuse is well done.
Weaknesses
1. It is unclear why the authors refer only to MAML (and its modification). If the paper introduces a new method for a few shots, the method should be compared with the state-of-the-art model using the gaussian process, transformers, and hyper networks. 
2. On the other hand, the paper can be seen as another MAML-based method analysis. But the contribution in the paper should be changed.
3. It is not clear why authors use randomly initialized neural networks. How does the method work on the pretrain model?
4. The fig. 1 should be better described.
5. Section 3.1 is not clear. Functions CalculateMask() and UpdateScore() are only mentioned and do not described in detail. 
questions: 1. How does the method work on the pretrain model?
2. Is it possible to train the weight of the network simultaneously with mask m?
3. It is a good way to compare original MAML and Meta-ticket with some architectures. Does it seem that MAML requires fewer parameters to solve a few-shot task than Meta-ticket?
limitations: The limitation section is well written.
review_title: null

Meta-review: metareview: This paper proposes a meta-learning algorithm which aims to find a subnetwork of a randomly initialized network, that is optimized for each task. Specifically, the authors propose to meta-learn the latent continuous parameters that correspond to a task-specific binary mask for each task, and provide empirical evidence and theoretical analysis which show that the method does rapid learning for each task rather than reusing the meta-learned features. The proposed Meta-ticket algorithm is validated on the few-shot classification tasks from various domains, and is shown to outperform existing meta-learning algorithms such as MAML, ANIL, and BOIL.

The reviewers in general were positive about the idea of finding a lottery ticket, or an optimal subnetwork for each task in a meta-learning scenario, and the discussion on rapid learning v.s. feature reuse. Also, they found the experimental validation as sufficiently extensive and the paper well-written.

However, there was a major concern regarding the work’s limited novelty over existing works that aim to learn a binary masking in a meta-learning, or multi-task scenarios, such as [Alizadeh et al. 2022], [Mallya et al. 2018]. Also, while not mentioned by the reviewers, [Lee et al. 20] cited in the paper also aim to learn a task-adaptive masking of shared initialization parameters. While the authors provided experimental comparison to ProsPr [Alizadeh et al. 2022], other methods were not discussed or compared against despite their relevance. There were also other minor concerns, such as missing ablation study on the hyperparameter, degraded performance with small neural networks, but they were mostly addressed away in the author responses. 

In sum, this is a well-written paper that provides a fresh view of the task-adaptation in the meta-learning in the perspective of lottery ticket hypothesis and feature reuse v.s. Rapid adaptation argument. However, due to limited novelty, discussion, and experimental comparison and analyses of the comparative study over existing works, I recommend a borderline accept for this paper in its current state. The authors are strongly advised to include discussion of [Mallya et al. 18], and experimental comparison against [Lee et al. 20] in the final version of the paper.

[Mallya et al.] Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights, CVPR 2018
[Lee et al. 20] Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Tasks, ICLR 2020
[Alizadeh et al. 22] Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradients, ICLR 2022
confidence: Less certain
Decision: Accept