Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a multi-task learning model with neural processes. The model is based on a hierarchical construction whereby each task is conditioned on global and local information. The paper derives a “hierarchical” ELBO for this model that is evaluated with MC sampling. Experiments are presented to validate the proposed model.
main_review: **Strengths:**

The general idea of the model seems to be a valid one for multitask learning, and to my knowledge, it seems original.

For the most part, the experimental section shows the strength of this model and the performance gains it brings against some other models in the literature. I particularly liked the last example on Brain Image segmentation.

**Weaknesses:**

I was a bit confused with the model construction. What exactly is the global variable $\mathbf{M}$? Is it simply the collection of the context datasets for all the tasks? Or is it a latent variable? If it is a latent variable, why do you opt for doing MAP estimation instead of placing a prior over it? I think this needs to be clarified further in the manuscript.

There are also some errors in the model construction section. The Gaussian likelihood is used to derive the generic multitask model. While this is a reasonable choice for regression tasks, it is a wrong assumption for classification tasks which are also considered in this paper. The inference section seems to have fixed this problem though. I suggest that section 3 is revisited to correct those discrepancies.

I would have liked to see a synthetic example exploring the properties of this model. The experiments jump straight to comparison on real-world datasets, but it would have been better to first start with a synthetic dataset to validate the modelling choices in Section (3).

Table (5) is missing confidence intervals. It also looks like the performance difference might not be significantly different from NPs for this example. I think this should be discussed.

Question: do the nested expectations in eq (11) cause any problems with the variance? How stable is the training of this model in comparison to standard neural processes?

summary_of_the_review: In my opinion, this work is original and has the potential to be an impactful contribution. However, in its current state, it is not ready yet. My main issues are the ambiguity in the model construction exposition and the lack of experiments that explore the model’s behaviour and properties.

**********
Edit: My score was updated to 6 after the rebuttal.
**********
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a multi-task neural process as a multi-task Bayesian model. 
main_review: Many Bayesian models have been proposed for multi-task learning. Why does the proposed multi-task neural process achieve superior performance? It is better to compare different models. 

The Gaussian likelihood is not suitable for classification tasks. Authors should consider other likelihood function.

In experiments, important baselines are missing. Authors should compare with MTL models such as cross-stitch network and MTAN, which can be modified to solve the multi-input multi-output setting. Moreover, multi-task Gaussian process should be compared.
summary_of_the_review: This paper proposes a multi-task Bayesian model. Compared with existing multi-task Bayesian models, I don't know why the proposed model is better. The proposed model is not suitable for classification tasks. Important baselines are missing in experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a multi-task neural processes approach in which function priors are derived in a hierarchical Bayesian inference framework to incorporate the shared knowledge into its context of the prediction function. Authors introduced a higher-level latent variable derived from the context data of related tasks to control the sharing of common knowledge between the tasks. Previous works used context data from its own task to generate the task-specific latent variables, whereas this work uses data from other tasks as well. The shared knowledge from all the tasks added to context of each individual task acts as inductive bias for predictions. Their experiment results show better performance of the proposed method than compared ones. 
main_review: Strength:
Even though rather limited, the proposed method shows better performance over compared methods for most of the learning tasks in their experiment. 

Weakness: 
The paper is poorly written, many places lack of clarity. (1) The description of the method is very confusing. In the main text, on page 4, it seems to suggest the \mu of and \Sigma of \psi are functions of m_l. However, in the network specifications in appendix, it seems \mu of and \Sigma of \psi has nothing to with m_l, because the input size of the \psi_1 network is 4096 but output size of h network is L. In addition, it is not clear why the h network is even needed. I did not see it is involved in either Eq. (12) or (13) to make predictions. (2) In section 3.1 it is mentioned that concrete formation of global variable M depends on the learning scenarios. However, it is not made clear anywhere in the paper how M is constructed from the context data. (3) It is unclear the multi-task setup in the brain image segmentation experiment. (4) How many Monte Carlo samples, i.e., N_f and N_a are drawn for the variational posteriors \psi and \alpha? And how they were determined?

Are networks \psi_1 and \psi_2 common networks shared among all tasks? Or are they task specific? If both are task specific, I do not see mathematically it would make much difference to train one single network comparing to the two that the authors chose to do. In other words, I do not see a clear value of proposed hierarchical design mathematically. More discussion on this would be very helpful. 

More empirical studies to support their claims are desired from both aspects of the methods to compare and learning tasks to test on. Most of the recently published multi-task learning methods were neither discussed or compared, for example Quo et. al., 2020, ICML, Sun et. al., 2020, NeurIPS, and Yu et. al., 2020 NeurIPS. The learning tasks used are limited to image recognition related tasks. The results could be more persuasive if tasks from diverse domains are included.   

summary_of_the_review: The paper is poorly written. The motivation of the proposed formulation is not clear. The empirical study is not adequate.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new multi-task neural process based on the classical neural process. The idea is to introduce additional global variables to share knowledge from different tasks. That is to say, a hierarchical bayesian model is constructed to link single-task neural processes together. The new model is able to handle multi-task applications contract to the classical neural process
main_review: The strong point is the extensive experiments. The authors have done extensive experiments to evaluate the proposed idea on various tasks. The results are very promising and significant. 

The weakness is possibly the weak theatrical contribution. The hierarchical model design and inference are straightforward. The authors did not compare with other possible solutions to show why this design is the best. The design of the original NP is guaranteed to be a valid stochastic process. It is pity that this discussion is missed for multi-task NP. Is multi-task NP still a valid stochastic process? is any marginal NP on a single task in multi-task NP still a valid stochastic process? 
summary_of_the_review: The paper is clearly organized and written. It is easy to follow. The results are very impressive. Although some important discussions are missed, this contribution of this work is good.
review_title: null

Meta-review: This paper extends neural processes (NPs) to the multi-task setting (MTNPs). The approach uses a hierarchical Bayesian construction, where the latent variables of an NP are conditioned on a set of global task-specific context variables. This allows the NP to share knowledge across related tasks.

There were a few issues raised in the reviews. Consistently, the reviewers noted that the writing could be improved. There were variables, like the context variables M, that lacked explanation. There was also confusion between the use of a Gaussian likelihood for classification vs regression. These were resolved with the author’s response and updated draft.

There were also requests for additional experiments and baselines: 1) a synthetic task, to which the authors included a 1D regression task. 2) More baselines against other multi-task methods, to which the authors included a comparison to Guo et al., 2020, MTAN, and multi-task Gaussian Processes.

Finally, there were questions around whether MTNPs are valid stochastic processes. This has been proven theoretically by the authors, albeit in the appendix.

Currently, this paper remains borderline. The main remaining criticisms are a) A desire for more experiments and analysis to highlight the particular strengths of the approach. b) That the approach is a straightforward extension of NPs, and may not be sufficiently novel. c) That the authors include more baselines from the recent multi-task literature (Yu et al. and Sun et al.). In the end it was determined that the paper does not quite meet the bar for acceptance. I think in future submissions, it would be worthwhile to further highlight MTNP’s performance in the low-data regime, where it particularly seems to do well, and to complete the full set of comparisons (e.g., Sun et al. and Yu et al.) that were requested by the reviewers.
Decision: Reject