Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper introduces a zero-shot task adaptation methodology by learning a meta-learner that reuses a latent representation of data points and tasks (Homoiconic Meta-Mapping). The proposed approach is applied to learn multivariate polynomial, a card game and the paper claims that such a method can apply in both supervised and RL settings.

+ves:

+ Task-to-task mapping is a relevant topic in the contemporary context, and the paper seeks to address this relevant problem.
+ The overall idea of considering winning-losing strategies in meta-learning is interesting.
+ The experimental settings of card games and polynomials are interesting to study the proposed setting, and the results are promising.

Concerns:

- A key issue is that the paper claims to be the first to propose a method for zero-shot task adaptation (pg 8, para before “Future Directions”). The paper seems to have missed papers such as (1) Zero-Shot Task Transfer, Pal et. al, CVPR 2019, and, (2) Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning, Junhyuk Oh et al, ICML 2017. In particular, the CVPR 2019 paper proposes zero-shot task transfer using meta-mappings. Citing these papers and comparing against them is essential, considering this paper addresses the same objective.

- The proposed methodology does not clearly detail how the zero-shot problem setting is used. Section F.2 (Appendix) gives the formulation of a meta-learner and zero-shot task a bit. However, how the meta-learner performs zero-shot task parameter regression is not clear. The notations used in mathematical equations are also not very clear, and a reader needs to swap between different sections to understand the proposed method. (Moving Sec F.2 to the main paper would be very useful. Similarly, moving Figs 5, 7, 8 etc. to the main paper would have helped greatly too.)

- How is the notion of a “task” defined? Are different classes of a dataset tasks (already done in [3]), or are different vision-related objectives as tasks (reference [1])? Are language-related objectives (such as: document classification, sentiment analysis, Named-entity recognition etc.) tasks? In an RL setting, do zero-shot tasks deal with a new environment? It would have been nice to see clearer definitions for the kind of tasks that this method is suited to.

- It was not clear on how the relationship between source tasks and zero-shot tasks are modeled or computed. Clearly, the performance of such a system depends on the correlations between the tasks, and these details are not clear, especially in generalizing the proposed method. The paper seems to correlate tasks and language-related hints; however getting a language hint is not always available, especially for datasets such as Taskonomy [1, see Refs below] - which is a dataset for multiple tasks.

- The zero-shot part is described using a winning-losing scenario, where we have a partial/side knowledge of winning of task_1 and the model can predict losing strategy. Now, what if:
We do not have any knowledge of the winning strategy (or losing strategy) of a zero-shot task?
Can we predict the losing strategy of a game, let’s say card game, from the knowledge of other games like: soccer, tennis, badminton etc. (the initial motivation of the paper). This is quite possible in the continual learning scenario (the paper mentioned some studies on continual learning). If so, then please comment on the negative transfers in that scenario and how it will affect the inference of the meta-learner.
The proposed method also mentioned language descriptions. In case of tasks that are not easy to describe (say, a vision task such as segmentation), how will the proposed method will compute the task-prior of seen-tasks and zero-shot tasks?
The paper mentions “winning-losing” strategy and the metanetwork gives losing strategy parameters. What if the scenario is the otehr way - given a losing strategy, we now have to perform the winning strategy (IMHO this has more value in the real world). Does the paper show the result on this, or am I missing something?

- Experiments are limited to synthetic data and card games, and it would be nice to see the validation of the proposed method on different domains. The notion of zero-shot tasks are also not clear for multivariate polynomial experiment and card-game experiment.

- Figure 1: please clearly mention in the caption what is the meaning and purpose of probes. It is not completely clear whether probes (also referred as held-out set In figure 1 caption) are taking part in training, or only at the time of validation?

- Shared Z vs. Separate task embedding: reuse of M (or, H network) network will only be possible if the input and output dimensions are same - i.e. the dimension of Z_output of M is the same as the dimension of Z_input of the M. What if  input and output dimensions are not the same?

Other minor comments:

* One general comment is: please mention the “Zero-shot” experiments more clearly (may be a separate paragraph or subsection), otherwise it is hard to follow.
* Figure 2(a) please remove the question mark (?) after Held-out. Similarly, Figure 5: Please complete the sentence “ Performing from …”. 
* Figure 1: (a)  Basic meta-learning: Please use some other color to highlight “(a)  Basic meta-learning”, the grey gives an illusion that the step ((a)  Basic meta-learning is not important.
* A term such as “Homoiconic” would have benefited from a formal definition, especially considering it’s not a commonly used term.
* The presentation is in general not well-polished, and needs revision.

References:
[1] Taskonomy, CVPR 2018, Zamir et al
[2] Zero-shot Task Transfer, CVPR 2019, Pal et al
[3] Learning To Detect Unseen Object Classes by Between-Class Attribute Transfer, CVPR 2009, Lampert et al.
[4] Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning, ICML 2017, Junhyuk Oh et al.

======= POST-REBUTTAL COMMENTS=========
I thank the authors for the response. Some of the queries/concerns in the review comments above were clarified. I am updating my rating to Weak Reject, but unfortunately am not completely convinced about accepting the paper in its current form for the following reasons:

- Thank you for pointing out the difference between this work and other recent efforts (Oh et al, 2017 and Pal et al, 2019), and updating the manuscript with this content. However, considering these methods have been proposed, it is only natural to ask that the proposed method needs to be compared against them, to judge the usefulness of meta-mappings as against other ways to achieve zero-shot task adaptation, at least on some common datasets/settings. The paper indeed does propose a new idea, but without a comprehensive comparison/evaluation, the inferences on its usefulness may not be conclusive without such studies.

- Considering methods solving the same/similar problem using other approaches have tested on real-world datasets, experiments on synthetic datasets alone seems limiting.  I agree and appreciate the amount of material that is already in the paper, but it is also important to have the correct material to make conclusive inferences that benefit the community.

I once again thank the authors for the efforts in the response, but just think the paper may benefit from revisions based on the comments shared here, to be more impactful/useful.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper presents a method for adapting a model that has been trained to perform one task, so that it can perform a new task (potentially without using any new training data at all—i.e., zero-shot learning). In some ways the presented work is a form of meta-learning or *meta-mapping* as the authors refer to it. The premise of the paper is very interesting and the overall problem is definitely of high interest and high potential impact.

I believe that the presentation of the proposed method can be significantly improved. The method description was a bit confusing and unclear to me. The experimental results presented were all done on small synthetic datasets and it’s hard to evaluate whether the method is practically useful. Furthermore, no comparisons were provided to any baselines/alternative methods. For example, in Sections 4 and 5 I was hoping to see comparisons to methods like MAML. Also, I felt that the proposed approach in Section 5 is very similar to MAML intuitively. This makes a comparison with MAML even more desirable. Without any comparisons it’s hard to tell how difficult the tasks under consideration are and what would amount to good performance on the held-out tasks.

In summary, I feel the paper tackles an interesting problem with an interesting approach, but the content could be organized much better. Also, this work would benefit significantly from a better experimental evaluation. For these reasons I lean towards rejecting this paper for now, but would love to see it refined for a future machine learning conference.

Also, the work by Platanios, et al. on contextual parameter generation is very relevant to this work as it tackles multi-task learning using HyperNetworks. It may be worth adding a short discussion/comparison to that work as it also considers zero-shot learning.

Minor comments:
- Capitalize: “section” -> “Section”, “appendix” -> “Appendix”, “fig.” -> “Figure”. Sometimes these are capitalized, but the use is inconsistent throughout the paper.
- “Hold-out” vs “held-out”. Be consistent and use “held-out” throughout.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors proposed to address the zero-shot adaptation of tasks by the defined and learned “meta-mappings”. Technically speaking, this work is built upon HyperNetworks, while its major contribution lies in the homoiconic embedding/treatment of a data point, a task, and a meta-mapping. Despite this fascinating and intriguing idea itself, I am still in doubt about its “real” power, considering that a few important references/baselines are missing.

Pros:
-	The ideas of introducing a meta-mapping and treating it similarly or the same as a learning a task itself are novel to me.
-	This work innovatively tackles the problem of zero-shot task adaptation, which is indeed challenging.
-	The case study of the proposed framework in the card game is interesting.

Cons:
-	The major concern for this work is its lack of discussion and comparison with state-of-the-art meta-learning baselines. 
    o	In fact, the meta-mapping this work learns is the relationship between tasks, and the meta-task the authors mentioned is a group of tasks. In this sense, I strongly suggest the authors to go through the recent work [1], and compare with it. Even the datasets constructed in that paper can be used as a benchmark to validate the proposed method in image classification.
    o	More recent work on explicitly learning the embedding of a task should be noted and compared, including [1][2]. The meta-mapping in this work is still based on embedding of a task, i.e., z_func, which is obtained by the function network. So is it possible to replace the embedding with recent SOTA algorithms?
    o	Other basic meta-learning algorithms like MAML[3] and many others should still be compared and discussed, especially in the basic meta-learning setting in Figure 2.
-	Many important details and ablation studies are missing, making the work less convincing.
    o	Why don’t merge the results Fig. 2 and Fig. 3 and compare them in a single figure, so that the contribution of meta-mappings can be shown. 
    o	How do you train the domain-specific encoder and decoder? Are they generalizable to a wide range of more complicated tasks like image classification? 
    o	It would be great to analyze the limitations of the proposed algorithm, especially for the case where a newly coming task belonging to an novel and unseen meta-task arrives. How insightful or predictable is the proposed zero-shot adaptation method in these cases?
    o	How do you obtain the lower-bound (solid line) and the upper-bound (dashed line) of the performance in the figures?
    o	Will the dimension of Z influence the performance? Since data points, tasks, and meta-mappings contain different amount of information, investigating the effect of the dimension of Z is key for the robustness of the method.
-	The paper is too expatiatory with many confusion notations and redundant parts, which make it quite difficult to follow.

References:
[1] Hierarchically Structured Meta-learning, ICML19
[2] TADAM: Task dependent adaptive metric for improved few-shot learning, NeurIPS18
[3] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

review_title: Official Blind Review #2

Meta-review: The authors presents a method for adapting models to new tasks in a zero shot manner using learned meta-mappings.  The reviewers largely agreed that this is an interesting and creative research direction.  However, there was also agreement that the writing was unclear in many sections, that the appropriate metalearning baselines were not compared to, and that the power of the method was unclear due to overly simplistic domains.  While the baseline issue was mostly cleared up in rebuttal and discussion, the other issues remain.  Thus, I recommend rejection at this time.
Decision: Reject