Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an encoder-decoder architecture with strong modeling capability on complex signals, which is capable of capturing the distribution of 1D, 2D and 3D signals efficiently. The main contributions are:
1. a framework that increases the capability of approximating complex signals (from 1D signal to 2D images and 3D shapes).
2. an efficient encoder that can significantly reduce computational cost.
3. a hierarchical decoder that learns multiple global latent variables for better approximation of global structure of a continuous function.
strength_and_weaknesses: **Strength**  
1. The proposed encoder structure is efficient in modeling complex signals especially for 2D and 3D data, compared with previous works (Attentive NP, Transformer NP, ConvNP, etc.)
2. The authors conduct extensive experiments on 1D, 2D and 3D data separately to demonstrate the effectiveness of the proposed framework on modeling different signals, compared with previous SOTA methods.
3. The authors also perform sufficient ablation studies to validate the design choices.

**Questions**
1. In fig.3 the results of VNP are pretty diverse, does this mean that the prediction is unstable/noisy?
2. For image completion comparison, are you using the same resolution(64 or 32?) for a fair comparison? Does the blurred results come from low-resolutions instead of a inferior method?

**Writing**
1. Typo: fig.4 "Our HINP"
2. I would suggest to ensure the tab/fig is on the same page as the text that mentions them. This will make reading easier. For example, fig.2, tab.2, fig.4.
clarity,_quality,_novelty_and_reproducibility: The paper is technically sound and novel. It is generally well written(except for some minor issues mentioned above) and easy to follow. It will be more reproducible if the authors can release the code and data.
summary_of_the_review: Given the above-mentioned strength (effectiveness over previous methods, extensive analysis, etc), my suggestion is that the paper is good for a publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present new architectural improvements to the Neural Process family, arguing that the costly self-attention mechanism designed to overcome context underfitting introduced in the work on Attentive Neural Processes (Kim et al., 2019) makes the method family unsuitable for large signals. Instead, the authors propose the use of a set tokenizer (Zaheer et al., 2017) which summaries several neighbouring context points into individual tokens, does reducing the complexity of the ANP attention mechanism. A second innovation are significant architectural changes to the decoder, introducing hierarchical latent variables, modulated MLP blocks as well as other changes. The authors evaluate their method on NP tasks and compare to a wide variety of other NP architectures. 
strength_and_weaknesses: `Strengths:`
- Neural Processes are interesting techniques for meta-learning INRs due to their reduction of the high memory cost of training, making this an interesting topic to investigate. Reducing the computational complexity of the ANP's self-attention is a valuable contribution.
- Results seem overall strong (Table 1, Figure 3), fewer FLOPs are required as stated (Table 3).

`Weaknesses:`
- While the Set Tokenizer is shown as a key innovation in the presented work, reducing the complexity of the attention mechanism, little insight is provided in terms of the cost that is paid by the summarisation of multiple context points. Table 1 shows that as expected, context reconstruction quality does decrease, but it is unclear how that decrease depends on the kernel size and stride (Section 4.1). No complexity analysis (in terms of $\mathcal{O}$ notation) is provided. 
- Due to a large number of architectural and algorithmic changes, the authors introduce a large number of additional hyperparameters to be tuned by a partitioner (number of hierarchical blocks/number of latent variables, $\beta$ (KL objective), kernel width/stride of set transformer). Reading between the lines, the method appears to suffer from potential instability "We only modulate some of the MLPs to preserve the training stability." 
- The manuscript in its current form has several clarity and quality issues making it currently unsuitable for publication. The submission appears rushed and with insufficient attention to writing.

`Questions:`
- To which extent are the improved results in Table 1 explained by the significantly larger models? What happens when baseline architectures are scaled to provide an equal comparison in terms of #parameters? 
clarity,_quality,_novelty_and_reproducibility: `Clarity & Quality:`
The clarity of the submitted manuscript is lacking, requiring the reader to consult several cited works to understand key contributions. Technical language is sometimes imprecise, using existing terms in unusual fashion or referring to certain related works by alternative names. Grammatical errors exist. Mathematical terms or unclear, notation is inconsistent. Examples:

- Section 4.2: "we use the modulated fully-connected (ModFC) layer (Karras et al., 2020; 2021) to adjust the inherent MLPs parameters". The authors should state the equations for those layers, saving the reader the effort to look this up. What are "inherent" MLP parameters? 

- (Garnelo et al., 2018b) should be referred simply as "Neural Processes" instead of "Latent Neural Processes", likewise (Garnelo et al., 2018a) is a "Conditional Neural Process".

- "ANP has troubles in processing complex signals that requires abundant context points as condition (e.g., image with high resolution), where the computational cost is very expensive." - Statements like this should be made more precise my stating the computational complexity of the ANP attention mechanisms (provided in those works). The proposed method should be compared in terms of computational complexity.

- Figure 1 is difficult to parse when encountered for the first time. For a first schematic overview of the model there is too much detail, which can be omitted given that Figure 2 shows the architecture in more detail anyway. 

- Section 4.1: "can reduce the number of tokens by 100 times". What is the image size in question?

- Equation (3): $q_{\phi}(\mathbf{z}|D_T)$ is not defined (merely called "conditional posterior encoder") - needs more explanation, $z$ is not bold in the sentence before.

- Equation (6): The sum is over $i$, while $i$ does not appear inside the summation.

- Equation (7): $z$ is again not bold, causing confusion (especially since $\mathbf{z}$ is redefined just before Equation (6)). Also, shouldn't the likelihood term also includes sampling and conditioning from all latent variable $\mathbf{z}_{1:K}$ used in the decoder?

- Introduction: "Moreover, for complex signal, [...]" -> "complex signals", or "a complex signal" 

- Section 4.1: "the computational burden is heavy and such framework is impractical" -> "such a framework".

- Section 4.2: "After achieving the low-dimensional latent variable" -> What does it mean to "achieve" a latent variable? Do the authors mean "sample"?

- Section 4.2: "they learn a single global" -> omit using "they" 

- Figure 3 caption: "tends to underfit the functions at some target locations" -> "underfitting" is usually a term reserved for context points in the NP literature. The graphic shows clearly this does not in fact happen, indeed the ANP shows better context set reconstruction (Table 1). The authors most likely mean "underestimates the variance".

`Reproducibility:`
    - Difficult to make a precise statement here without a concrete attempt. I'd urge the authors to provide an implementation in the supplementary mterial
summary_of_the_review: Overall an interesting idea with the potential for publication, but currently unsuitable due to serious Clarity & Quality issues. Will consider raising my score if the authors significantly improve those aspects during a rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an auto-encoder based pipeline to achieve versatile neural process, which consists of a bottleneck encoder and modulated MLP based decoder. The bottleneck encoder aims to produce fewer and informative context tokens while the decoder hierarchically learns multiple global latent variables and the uncertainty of a function to improve the capability in modeling complex signals. While the developed method follows the existing paradigms quite a bit, it shows practical improvements compared to chosen baselines under three different tasks.
strength_and_weaknesses: Strengths

(1) This paper is generally well-written and easy to follow, except some points. I can understand most statements easily. For some improvement suggestions, please see the weakness part below.

(2) The authors present consistly improved results over chosen baselines under three different tasks.

(3) Most of the technical elements are built on existing literatures, providing solid effectiveness foundations.

Weakness

(1) The novelty of this submission seems to be limited. For example, using self-attention layers && set tokenizer (in Sec. 4.1) and hierarchical latent codes (in Sec. 4.2) to improve results is not new. Although the combinations of these strategies produce appealing results, the authors do not show specific design insights, making the proposed method look a bit ad-doc.

(2) There are no discussions on the limitation and failure cases.

(3) The current developed framework is more complex than previous methods. Given some practical improvements, I am curious about how much time it needs for the training and testing? Please report the training and inference speed as well as corresponding experimental environment for more comprehensive evaluations.

(4) For the baseline comparisons, why do not the authors make evaluations over optimization-based implicit representation modeling methods (e.g. SIREN, Fourier Feature Network)? I look forward to seeing these empirical comparision results.

(5) In terms of experimental results, the used datasets are too simple. How about the results produced by NeRF's synthetic datasets, the complex shape sets used in SIREN and NGLoD, et al.? Please use more complex datasets to prove the technical advantages.

(6) Some minor presentation issues:

A. In Sec. 4.2, the functionality of cross-attention is unclear. Why should we use the cross-attention here?

B. In Sec. 4.2, why multiplying the KL term with a small weight can better capture the uncertainty of function distributions?
clarity,_quality,_novelty_and_reproducibility: Generally, this paper is easy to follow with clear descriptions. However, as for me, it is limited in technical novelty and somewhat looks like an incremental revision over existing approaches. Please see comments above.
summary_of_the_review: Although this paper yields some interesting empirical results compared to some baselines, the proposed method builds mainly on top of previous techniques without enough technical contributions. Additionally, it is also weak in the experimental part, including lack of evaluations over challenging datasets and optimization-based methods. However, I am not an expert in this area, so I will not object if the other reviewers tend to accept and think it has already met the bar of ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new architecture for Neural Processes named Versatile Neural Process (VNP). This new architecture is build as an improvement over previous Attentive Neural Process (ANP) with two main modifications:
1. The context points are pre-processed by a encoder module built using set convolutions and self-attention layers. This module summarizes the (potentially large) set of context points into a much smaller set of context tokens. This allows the model to scale to much larger sets of context points than ANP.
2. The decoder part of the NP is augmented with a hierarchy of random variables (as shown by (Child 2021), this has a large impact on expressiveness to VAE-like models). These variables are used to module the prediction in an iterative way.

These two modifications together are empirically shown to significantly improve the capacity of VNP to model complex and varied functional distributions, as well as computationally scale to large inputs such as super-resolution tasks.
strength_and_weaknesses: **Strengths:**
- The proposed architecture and its training procedure is described with extensive details.
- The proposed VNP is empirically compared to many other models from the literature on various tasks.
  - The improvement on 1D tasks are impressive, showing how VNP is able to capture the variability of the function distribution much more effectively than previous models.
  - The comparison of computational cost on the image reconstruction task clearly shows the gain from the added encoder, making the prediction cost almost independent on the input size.

**Weaknesses:**
- The tasks used to asses the model quality are artificial problems. This is an issue shared by the other NP models from the literature, but as the model family is maturing it would be interesting to asses its applicability to real world problems. After reading the paper I don't have a clear view of when a practitioner would want to use a VNP.
  - In particular, in the context of the super-resolution task, VNP is compared to other NP models, but not to the general state of the art of super-resolution models, which also contains models able to do arbitrary-scale super-resolution (such as *Learning Continuous Image Representation with Local Implicit Image Function*, Chen, Liu & Wang, 2021, cited in the paper).
clarity,_quality,_novelty_and_reproducibility: The paper is clear and well written. It introduces a novel NP architecture by intelligently combining previous work from the literature (ANP, set tokenizer and modulated layers).

The paper provides details about the model structure and experimental setup which seem sufficient to reproduce similar experiments, but the exact structure of the models (number and size of layers, etc..) are not given, even in appendix.
summary_of_the_review: This paper introduces a new iteration on the design of NPs which significantly improves the capacity of the model to capture variability in the function distribution as well as its scalability with regard to the number of context points.

While the application cases for NPs remains rather nebulous to me, I believe this is a good paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a new architecture for versatile neural process. The new approach shows practical improvements over baselines. The reviewers generally agree that the method is interesting and convincing, and the results are strong. The AC agrees with the reviewers and recommends accepting the paper.
justification_for_why_not_higher_score: There are typos and the clarify of the paper needs improvement.
Decision: Accept: poster