Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The submission proposes a meta-learning algorithm attuned to the hierarchical structure of a dataset of tasks. Hierarchy is enforced in a set of synthetically-generated regression tasks via the data-sampling procedure, which is modified from the task-sampling procedure of [1] to include an additional source of randomness corresponding to which of a set of cluster components task parameters are generated from. The authors propose to adapt the model-agnostic meta-learning algorithm (MAML) of [1] to reflect this hierarchical structure by either observing (Section 4.1, FixedTree MAML) or inferring (Section 4.2, LearnedTree MAML) an assignment of tasks to clusters at each step of the inner loop (task-specific adaptation phase) of MAML; if tasks belong to the same cluster, the correspond task-parameters receive the same update at that step (in particular, the update direction is averaged). It is assumed that there are increasingly many clusters at each step, so that task-specific parameter updates are increasingly granular.

##### Strengths:
1) **Clarity**: The experimental setting and exactly how the data-generating process relates to the proposed algorithms are clearly described.
2) **Significance**: Results on the hierarchically structured synthetic regression task datasets demonstrate that {Fixed|Learned}Tree MAML: is at least as good as MAML, and often outperforms MAML; that it learns more efficiently than a MAML in terms of the cumulative number of datapoints observed; and that both MAML and {Fixed|Learned}Tree MAML outperform a naive baseline.

##### Weaknesses:
1) **Significance**: Since the evidence provided in favor of the proposed algorithm is in the form of an empirical evaluation on a synthetically generated dataset, the present impact of the algorithm is limited. In particular, there is no evidence that (i) the algorithm works for larger and/or more complex datasets; and (ii) that natural datasets of interest to the community exhibit a hierarchical structure analogous to the synthetic datasets presented in the submission.
2) **Novelty**: The algorithm modifies and combines previously introduced components: the MAML algorithm of [1]; the online top-down clustering algorithm of [2], and the task-similarity-as-gradient-similarity approach of [3].
3) **Clarity**: Specific details surrounding the relationship between Algorithm  1 and Algorithm 2 are insufficiently discussed: 
  i) Algorithm 2 as it appears in the text is very similar to Algorithm 1 (The OTD algorithm) in [2] with the exception of the new hyperparameter $\xi$, and introduces new symbols that do not appear elsewhere in the text. It is therefore not sufficiently adapted for clarity in the context of this work.
  ii) Whether Algorithm 2 acts as a strict subroutine of Algorithm 2 is not stated. I believe it is not because the clustering decision for a new task relies on tree structures that are "generated for a training batch," although what a "training batch" refers to is not clear. Similarly, how the "online"/"offline" distinction in the context of the clustering algorithm fits into the training/evaluating setup borrowed from [1] is not made clear.
  iii) How exactly the task-similarity approach of [3] is employed in Algorithm 2 is not made clear. The only mention of the use of [3] is briefly around Eq. (8) before the main algorithm (Algorithm 1) is introduced, and Algorithm 2 only refers to a generic "similarity metric" (as in the original work, [1]).

##### References

[1] [Finn, Chelsea, Pieter Abbeel, and Sergey Levine. "Model-agnostic meta-learning for fast adaptation of deep networks." arXiv preprint arXiv:1703.03400 (2017).](https://arxiv.org/abs/1703.03400)

[2] [Menon, Aditya Krishna, Anand Rajagopalan, Baris Sumengen, Gui Citovsky, Qin Cao, and Sanjiv Kumar. "Online Hierarchical Clustering Approximations." arXiv preprint arXiv:1909.09667 (2019).](https://arxiv.org/abs/1909.09667)

[3] [Achille, Alessandro, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, and Pietro Perona. "Task2vec: Task embedding for meta-learning." In Proceedings of the IEEE International Conference on Computer Vision, pp. 6430-6439. 2019.](https://arxiv.org/abs/1902.03545)
review_title: While appropriately taking into account task similarity & structure in meta-learning and related fields is an important open problem, the scope of the paper is limited to a specific combination of existing algorithms on synthetic datasets. The submission also needs some work to make its methodology clearer.

final_score: rating: 4: Ok but not good enough - rejection
review_content: What is this paper about, what contributions does it make, what are the main strengths and weaknesses?
This paper proposed a learning algorithm of meta-learning: TreeMAML, to share information across tasks in meta-learning models. The paper compared the results of TreeMAML with MAML and the Baseline on SinusRegression Task and Linear Regression Task.

The main concerns are:

1.	Some statements in the introduction session are improper. 

1.1	“there is still a lack of methods for sharing information across tasks in meta-learning models, and the goal of our work is to fill this gap.” See references:
a.	Invenio: Discovering hidden relationships between tasks/domains using structured meta learning. 2019. 
b.	Hierarchical meta learning. 2019. 
c.	Hierarchically structured meta-learning. 2019. 
d.	Learning to propagate for graph meta-learning. 2019. 

1.2	"However, these algorithms are not model-agnostic." Most of the related models are model-agnostic as long as replacing the task representation module. 



2.	The contribution of this work is supposed to be emphasized. The innovation of this paper seems insufficient. 

2.1	Yao et al. (2019). also aims to utilize the task relation and apply gradient-based adaptation methods. And the model structure presented in figure 1 is similar to the model framework in Yao’s paper.  

2.2	As claimed in this paper, the hierarchical clustering algorithm, which is an essential part of the proposed model, is introduced with minor modification from Menon et al. (2019). 

3.	As claimed in this paper, the proposed model lacks scalability since it works well only in the tree-structured tasks. 

4.	The baselines exclude many relevant works such as the papers listed above.

review_title: Official Blind Review #3

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################

Summary:
The paper studies a simple modification of MAML to address the problem of meta-learning hierarchical
task distributions. It is based on the assumption that learning tasks by gradient descent may benefit from gradient sharing, across similar tasks. The authors convert the problem of task transfer to gradient clustering by considering the point that the similarity of tasks can be measured by the similarity of gradients.

##########################################################################

Reasons for score: 

 
Overall, I vote for rejecting this paper. Unfortunately, I found the novelty of the paper very low. I believe that this paper is like an incremental study of exist work like MAML[1] and HSML[2].
[1]https://arxiv.org/abs/1703.03400
[2]http://proceedings.mlr.press/v97/yao19b/yao19b.pdf

 
##########################################################################
Pros: 
- The paper is clear and cites the most relevant research studies and papers.
- The experimental results presents the promising performance of the method.
- It is a strength point of the paper that the authors address both foxed tree and  learned tree structures.

##########################################################################

Cons: 
- The novelty of the paper is very low. The authors refer to [2] in the section 2 of the paper and mention that it is not task agnostic.
a) I believe HSML[2] is also built on MAML and not sure about this claim.
- Lack of extensive experiments:
a) The paper does not include any experiment or discussion comparing the proposed method with HSML[2] or other exist method. I would suggest the authors to consider running more experiments.
b) The time complexity of the method has not been discussed.
 
##########################################################################

Questions during rebuttal period: 

 
Please address and clarify the cons above 

 
#########################################################################




 
review_title: Model agnostic meta-learning on trees

final_score: rating: 3: Clear rejection
review_content: #### Summary
- In the context of gradient-based meta-learning for few-shot learning, the authors propose TreeMAML, an algorithm that leverages the existence of a tree structure in a task distribution in order to pool inner-loop gradients between tasks. More specifically, the inner-loop corresponds to a level-by-level traversal from root to leaves, where at each step, task-leaves that are children to a common node in the current level average their gradients for this step's update. 
- The authors consider two cases: one in which the ground-truth tree structure is known (Fixed TreeMAML) and one in which it is unknown and must be discovered online (Learned TreeMAML). The authors extend a hierarchical clustering algorithm from prior work for this purpose. 
- The authors evaluate TreeMAML in three scenarios: sinusoid regression (Finn et al. 2017), linear regression, and mixed regression (Yao et al. 2019). TreeMAML compares favorably against MAML and a naive multi-task learning baseline. 
- Interestingly, Learned TreeMAML also consistently outperforms Fixed TreeMAML.


#### Strengths
- The authors investigate an important and under-considered problem in meta-learning: how to leverage structure within a task distribution. 

- The proposed TreeMAML algorithm is conceptually simple.

#### Weaknesses
- The motivation for the TreeMAML algorithm is very weak. Yes, averaging gradients across tasks might decrease variance, but presumably always increases bias. This crucial trade-off is not even mentioned.

- The fact that Learned outperforms Fixed is troubling. It suggests that TreeMAML is not properly leveraging the ground-truth task hierarchy. Dissecting Learned to look at the tree structure it proposes would help diagnose this issue.

- This is a purely empirical paper which only presents results in toy regression settings. More comprehensive empirical evaluation is needed, e.g. the image classification benchmark proposed in Yao et al. (2019), which is perfectly suitable for this paper (and indeed the authors took Experiment 3 from this work). 

- The comparison between TreeMAML and MAML might not be very fair. MAML is artificially constrained to use the same number of inner-loop steps as the depth of the task tree. Since MAML makes no assumptions about the task tree, this should be a tunable hyperparameter.

#### Recommendation
- I currently recommend a clear reject (3). Given the weaknesses outlined above, this submission does not meet the bar for acceptance.

#### Questions
- How is the Baseline model trained? Its MSE in Table 2 is uncommonly high.

#### Minor suggestions
- Please fix the numerous typos throughout the submission. Just in the first paragraph: inappropriate capitalization of multi-task and meta-learning; "The field of of".
- K is used to denote the number of inner gradient steps in Alg. 1, but the number of datapoints per task in Sec. 4.1.
- The num_shots=3 case in Fig. 3b directly contradicts the caption.
review_title: Interesting ideas, but execution is weak.

Meta-review: The paper proposes a variant of MAML for meta-learning on tasks with a hierarchical tree structure. The proposed algorithm is evaluated on synthetic datasets, and it compares favorably to MAML. The reviewers identified several significant weaknesses, including: (1) the experimental evaluation is limited, and it only includes small synthetic datasets; (2) the proposed algorithm is incremental over MAML. The reviewers agreed that the paper cannot be accepted in its current form. I recommend reject.
Decision: Reject