Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper has a very good motivation of combining neural processes (NP) and a mixture of experts (MoE) with the hope to further strengthen the power of NP as MoE usually does with other architectures in the application of context-based meta-learning. To enable a collection of experts, a single global latent variable is replaced by a collection of latent variables, which represent MoE NPs. 
strengths_and_weaknesses: Strength 
1. Apart from motivation, the paper has clear explanations of step-by-step derivations.
2. Experiments show strong improvements over the chosen baselines. 

Weakness
1. One important question in practice is scalability. Naturally expanded from NP, it would be good to compare it with at least NP since the model is more and more complex as more latent variables are injected. Also, it’s not clear (as also mentioned in 5.5) why an increasing number of experts would not help after 5? 

- So, I would suggest doing some expert-specific studies about mean/variances, learned representation, and system-related statistics to see the difference between them. One hypothesis is that the model collapses somewhere that only a few experts actually learn and thus increasing the number of experts would not help. This can vary by tasks/data distributions as well. 

- Similarly for Latent Variables ablation in 5.5, I would suggest doing a per-task, per-expert analysis to see if it matches the intuition on whether each expert is specialized for each (group of) task(s), which would further solidify the intuition to replace a global variable with different local variables.   

2. Despite a very good motivation and setting, there might be a need to compare with some other gradient-based MoE performance because MoE naturally enhances models’ complexity (as in the same case here with the introduction of more latent variables) for more power, in exchange for being more complex and other costs such as communication and/or memory. Likewise, it would be good if we also have some comparison with some other MoE baselines such as GShard, Deepspeed MoE, …

3. No code is provided. 

questions: Line 66/67 seems to be overly stated, implying gradient-based optimization is slower than NP, which might not be true. Gaussian Processes have cubic cost without proper modification, and NP marries GP and NN to inherit the fast training. Consequently, it is not clear whether NP is strictly faster than NN, at least based on this hybrid intuition. Please elaborate or explain why your claim is so. 


It seems for simpler tasks, the model performs really well but with more difficult tasks, the variance is much higher (Fig 7), is it because of variational and Monte Carlo approximations over so many latent variables? 


Similar to the scalability point in Weakness above: although the authors already presented a high performance in difficult settings, e.g. Humannoid, it would be more practical for this solution if you could also demonstrate a higher-scale experiment in which there are a big number of tasks, maybe meta-learning for multilingual MT. 


See also in Weakness above. 

limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the problem of learning-to-learn a new task $\tau$ from a dataset $D_{\tau}$ of labeled pairs $(x_1, y_1), ..., (x_n, y_n)$. The authors propose to do so using meta learning with neural processes (NPs), trained via variational inference. They propose a new type of NP model that uses a multi-modal mixture density as a prior, called Mixture of Expert Neural Processes (MoE-NPs).

The problem with vanilla NPs is that they use a single gaussian latent variable z to specify an entire task. As result they have limited expressiveness when trying to learn functional priors over tasks from unknown distributions. The authors propose to resolve this issue by using a mixture prior with K gaussian distributions and latent variables $z_{1:K}, e$ where $e$ is a categorical mixture assignment variable.
strengths_and_weaknesses: The paper writing is generally clear and thorough. Although I was unfamiliar with NPs before reading the paper, I believe that the proposed extension provides an original and significant contribution to the field based on the provided theory and experiments.
questions: There is an important choice that has to be made for the MoE-NP model, which is the number of mixtures K. In all of the experiments the authors take this number as a given (K=3 for Acrobot, K=2 for CIFAR-10), but I assume there is a lot of tuning and optimization that goes on behind the scenes to settle upon a value that works well. I would suggest that the authors say a little bit about this tuning procedure and what it looked like for the experiments.

Pretty much all of the implementation details of the model are left to the appendix, which is non-standard. I suggest finding some room in the main paper to say at least a little bit about the implementation you're working with.
limitations: Limitations are adequately addressed
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new Neural Processes family, Mixture of Expert Neural Processes (MoE-NPs). While NPs represent each task as a single vector representation sampled from Gaussian distribution, MoE-NPs use discrete representation to select which representation will be used for each data. Therefore, MoE-NPs have more flexibility than other NPs and it is shown in this paper empirically from 1D sinusoidal regression task to Meta Reinforcement Learning tasks.


==========================
The authors addressed my concerns enough, I increase my score.
strengths_and_weaknesses: Strengths:
- MoE-NPs take the idea of Mixture of Expert to NPs and show better expressiveness than other NPs.
- Equations are derived in appendix, which helps to understand.
- Including ANP and Conv-NP, they evaluated MoE-NPs with a wide range of baselines comprehensively.

Weaknesses
- I can understand MoE-NPs can have more representative power than NPs, but I am not sure when we need it. The toy regression task is to show the strength of this method, but except that, I cannot find the killer applications for MoE-NPs. MoE-NPs shows more stable results than others for image completion in Figure 4, but the y-axis range is quite small, so I am not sure we can say MoE-NPs is necessary for that. For RL, MoE-NP didn't show a significant performance gap from baselines.
- In Figure 8, it is also weakness that overestimated experts hurt the performance. I think it is because by representing a single underlying function with more than 1 expert, the model misses the underlying patterns.
- In section 4.2, notations are quite confusing (e.g., $q_{\pi 2,1}$ and $p_{\pi 2,2}$ are the variational posterior and prior for categorical variable, why $(2,1)$ and $(2,2)$?). They are not well described before this section, so it is hard to understand. 
- For toy regression task (section 5.2), two functions look no variations without the noise. For example, in [1], they made variations on amplitudes or phase, but as described in section 5.2, this task doesn't have those variations to learn the meta-learning framework. I don't think that it is enough to validate those meta-learning models.

[1] Finn, Chelsea, Pieter Abbeel, and Sergey Levine. "Model-agnostic meta-learning for fast adaptation of deep networks." International conference on machine learning. PMLR, 2017.
questions: - In Table 4, ANPs results are same for the different number of given contexts. Is it right? or typo?
- As described in the above session, I am not sure when MoE-NPs is strong. When we need to use different $z$ in a task?
- In section 5.2, the amplitudes or phases of $sin$ and $cos$ function, is it right?

limitations: I think the authors already addressed the limitations of their model adequately, but more clarifying why it is required for some cases will make this paper more concrete.
review_title: null

Meta-review: metareview: The paper introduces a mixture of expert prior in the neural processes (NP) models as a way of improving the expressibility of the prior and posterior. The paper is well motivated, written well, and the method is reasonable and sound. The experiment results are also comprehensive and convincing. The authors well addressed the reviewer's questions in the rebuttal. All reviewers agreed on accepting this paper.
confidence: Certain
Decision: Accept