Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes Implicit Process Meta-Learning (IPML) where each task is represented as a continuous latent vector $\mathbf{z}$, and corresponding data points are described as function values evaluated at an implicit process conditioned on the task latent vector $\mathbf{z}$. To conduct the intractable inference, a stochastic gradient Hamiltonian Monte Carlo (SGHMC) algorithm is employed. A VAE-like network called X-Net is trained simultaneously to generate synthetic tasks from the task latent vectors. The experimental results demonstrate that the proposed algorithm shows decent performances on few-shot classification tasks, and the task latent vectors indeed represent a meaningful space of the tasks on which measuring distances between tasks and detecting outlier tasks are possible.

Overall I find this paper is clearly written. Various experiments are conducted to demonstrate the benefits of the proposed model. Personally, I like the implicit process framework, and I guess this is a nice application of the idea to meta-learning. 

Since the performance of the proposed algorithm is not overwhelming, I presume the main advantage of the proposed algorithm is its ability to model a proper latent space of tasks.  My first concern is that whether this is a property only for the proposed IPML. For instance, consider a prototypical net. Although not being as principled as IPML, a prototypical net can also represent a task via prototypical vectors and consider them as a latent representation of the task (e.g., just take the average of all prototype vectors to get a single representative of a task).  Is this space completely meaningless so measuring the distance metric doesn't reflect the semantic relationship between tasks at all?

Also, as far as I understood, the ability to construct a meaningful latent space of tasks is not solely from the implicit process itself but aided by the additional X-Net learning a generative model over the data points in the tasks. What if we conduct measuring the MMD between tasks for the model without X-Net (not between the generated tasks and an existing task, but between existing tasks semantically apart)? Moreover, how the prototypical-like model with a generative model attached compare to the proposed one on synthetic task generation? The reason why I'm concerned about this is training a generative model and generating an image from it is usually quite expensive. For instance, one can easily imagine that training X-Net for miniImageNet or tiredImageNet and generating images from it might be effective.

Some minor questions
- Figure 3 shows the shift in the latent space when tasks data are manipulated. But is it really demonstrate the semantic difference between tasks? I guess a representation from an ordinary classifier might also show this because the input "images" were shifted using non-trivial operations. I guess this point may be made clear by having a look at the representations of semantically different tasks (e.g., classification of different classes) without being modified at the image level.
- For a binary classification task (Figure 4), when generating a task from a task vector $\mathbf{z}$,  as far as I understood, each data $\mathbf{x}$ is generated conditioned on $\mathbf{z}$ and then corresponding label $y$ is generated. Are the generated data is well balanced between positive and negative classes?
- Have you considered using SGLD instead of SGHMC? If so, did you observe any empirical degrade in the performance?

review_title: A solid paper but unclear motivation

final_score: rating: 6: Marginally above acceptance threshold
review_content: === Summary ===

The paper proposes a meta-learning method based on implicit process (IP) framework in which each task is represented by a latent vector. The IP setup for meta-learning seems identical to that of Neural processes [1]. In that, the key challenge for adaptation to a task based on a context/support set $(X_c, Y_c)$ is inferring the distribution $p(z|X_c, Y_c)$ over latent vectors. While [1] use amortized (variational) inference with a variational Gaussian distribution to approximate the true $p(z|X_c, Y_c)$, the paper proposes to use stochastic gradient Hamiltonian Monte Carlo (SG-HMC) for sampling latent state vectors from $p(z|X_c, Y_c)$.

In the experiment section, the paper compares the proposed method (IMPL) against various previous methods, mostly gradient-based meta-learners, on common benchmarks, demonstrating competitive results. In addition, the paper provides a detailed inspection of the latent space and argues that IMPL could be used for active task selection and synthetic task generation.

=== Main argument ===

The IP meta-learning setup of the paper is well-known in the meta-learning community (e.g. [1]) and thus not a contribution. Unlike to previous work (e.g. [1]) which uses the IP framework for meta-learning, the paper proposes to use SG-HMC instead of amortized (variational) inference for approximating p(z|X_c, Y_c), resulting in a new, sound meta-learning algorithm. This can be considered the methodological main contribution of the paper.

The experiment section features standard, well-accepted meta-learning benchmarks and demonstrates that IMPL is not only theoretically sound but also performs well in practice. The analysis and visualization of the learned latent task representation is interesting and provides nice qualitative evidence that proximity of task representation vectors in the latent space reflects the human-perceived task similarity to a fair degree. 

Unfortunately, the paper’s experiment section does not address some of the core questions: How does IMPL compare to Neural Processes (NPs)? What are the pros and cons of replacing the amortized (variational) inference with SG-HMC? E.g. How much performance increase does the arguably much more complex and computationally demanding approximate inference method buy us? Since, the key contribution of the paper is replacing the approximate inference method of NPs with SG-HMC, an empirical in-detail comparison with NPs would strengthen the paper a lot!

In the related work section, the paper argues that a limitation of NPs is the Gaussian variational distribution in the latent space. However, since the latent space is meta-learned and thus quite arbitrary - this is not a very convincing argument. Empirical evidence for the hypothesized shortcomings of NPs would be much better.

Furthermore, it remains unclear why/how active task selection could be relevant in any real-world setting. Motivating the setup with a real-world use case would strengthen this aspect of the paper. The real-world risk detection experimental results in the appendix are nice and interesting. Putting more of the results in the main part (there is still some space) and briefly discussing them could give a nice argument for the benefits of the latent task representation inherent in IMPL - i.e. it provides a straightforward way for identifying tasks that are very dissimilar from the rest - excluding such outlier tasks may increase performance.

=== Overall assessment === 

In the current form, I see the paper slightly above the acceptance threshold. The proposed meta-learning algorithm based on IPs is sound and the extensive experiments demonstrate that it works well in practice. However, the only innovation of IMPL over Neural Processes (NPs) is replacing variational inference by SG-HMC. Thus, the algorithmic contribution is limited. What the advantage of IMPL over NPs is, has not been addressed sufficiently. If a proper experimental comparison to NPs is added to the paper, I am happy to increase my score.

[1] Garnelo, Marta, et al. "Conditional neural processes." arXiv preprint arXiv:1807.01613 (2018).

review_title: Solid paper - Though, better comparison to Neural Processes is needed

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an efficient meta-learning approach using implicit processes. Specifically, authors represent each task as a continuous latent vector and use expectation-maximization algorithm to perform meta-learning. The E step performs task adaption using stochastic gradient Hamiltonian Monte Carlo sampling method, while the M step optimizes meta-learning objective using these samples. Their framework can measure principled distance between tasks by maximum mean discrepancy (MMD) and generate synthetic tasks by task-dependent distribution. Finally, the authors validate their proposed framework on several benchmark datasets and real-world datasets. The novelty and originality of this paper is good by proposing new ideas and methods. In addition, the paper is well-organized and clearly written. We can quickly get to know what problem they are trying to solve, how they solve and what their results are.

Pros: 
1.Representing each task as a continuous latent vector is very suitable when ones need to measure the task similarity, synthesize new tasks or actively select a task without the assumption of known task contexts.
2.The expectation-maximization algorithm based on the stochastic gradient Hamiltonian Monte Carlo sampling method can mitigate the enormous computation overhead by using first-order approximation without second-order derivative.
3.Authors demonstrate the effectiveness of this IPML on benchmark datasets and real-world tasks.

Cons:
1.The results of IPML in Table 2 do not outperform the baseline methods. I think authors could provide more analysis or reasons on results.
2.Authors should add more ablation study and analysis on why the proposed method would work. For example, how much is the contribution of representing each task as a continuous latent vector? How much is the contribution of the expectation-maximization algorithm?
3.Apart from the performance in the experiment, authors should also mention the efficiency of the IPML compared with other baseline methods. For example, the run time of the algorithm on computational expensive tasks.

review_title: We review the proposed method and the experiment results

Meta-review: This paper sits at the borderline: the reviewers agree it is a well-written and interesting paper, but have concerns about efficiency as well as a comparison with the neural process (the authors did include a revision with this comparison, though the numbers they report are worse than in the original neural processes paper on the same experiment). Ultimately, this paper probably requires another round of reviews before it is ready for publication.
Decision: Reject