Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper develops a special kind of neural network architecture that the authors call SPIN, semiparametric inducing point networks.  This architecture takes a dataset as input and using a 
“dataset encoder” outputs a latent set of “inducing points” which one could imagine either as pseudo-examples representing the larger dataset or summary statistics describing the data.  These inducing points are incorporated into a prediction network via cross-attention layers that effectively compute dot products between a query and the inducing points.  The authors demonstrate the efficacy of this approach empirically on the UCI data, a meta learning setup and a genotype imputation problem.
strength_and_weaknesses: Strengths:
The paper is clear and well written.
The proposed model seems intuitive, scalable and effective.
I like that the authors found multiple problem domains where their proposed approach would outshine traditional GP or deep learning approaches.  Genomic imputation is neat and seems like an important real world application.
The method seems to work well in the experiments presented.
The GPU memory required by the method seems clearly less than the stronger baseline NPT.  

Weaknesses:
A major aspect of the neural process literature is that the models are a stochastic process, i.e. they induce a distribution over functions conditioned on some observed data.  This paper completely ignores that part and considers only predictive accuracy.  For example, a major claimed advantage of neural processes is that they can provide a high quality estimate of uncertainty away from the data.  This paper completely ignores the “processes” part of neural processes and evaluates the models as if they’re standard deep networks on standard deep learning benchmarks (agreed genomic imputation is non-standard and more exciting). 
The treatment of existing work is a little sloppy.  I’m not sure why inducing points and deep kernel learning are attributed to Wilson & X.  There’s a deep literature on both subjects and these papers are one contribution of many in each subfield (neither the first, last or SOTA).
The choice of baselines seems quite weak.  Particularly, UCI has been completely and utterly annihilated by Bayesian deep learning papers.  Take a look at all the papers citing Bayesian dropout (Gal & Gharhamani).  The fact that the baselines here are: a method from 2001 - GBT, K-Nearest Neighbors and “Multilayer Perceptron” seems quite suspect given that one can just extract the numbers from recent papers.  Why are all the baselines from 20 years ago or more?  That is compounded by the fact that only a ranking is provided, so a reader can’t directly compare to more recent literature.  I found the results in the appendix and confirmed that 1) in general they’re not as strong as dropout (which is taken to be a straw man in many papers) and 2) the results often don’t seem statistically significant.  Could the authors provide some more context about why they chose these baselines and the significance of the ranking results?
The ranking results are just not statistically significantly better than NPT.  I think it’s ok to say they are competitive with NPT but require less than half the memory, but it doesn’t seem justified to say that they are better on average.
If the claim is that SPIN / IPNP is faster, then plot with walltime as an axis.

clarity,_quality,_novelty_and_reproducibility: The paper is very well written and easy to follow.  I think the presented architecture is novel, but I’m a little unsure given how many transformer / attention variants there are currently.  So I would defer to other reviewers on novelty.  The authors provide detail about their architectures and in the appendix they detail hyperparameters, so I’d feel confident I could reproduce the experiments with some effort.

Nitpicking:
Inducing points are not attributable to Wilson et al.  That’s a strange citation to have. You should cite e.g. Snelson & Ghahramani, 2005.  I’d really like to see a better treatment of the existing literature on inducing point approximations.

“Inference” is an overloaded term in deep learning and statistical inference.  I.e. they mean different things in deep learning and non-parametric modeling.  I’d just avoid using it altogether.


summary_of_the_review: Overall, I think this paper represents an interesting innovation that seems novel and is well described.  I think the paper missed the mark a bit in terms of formulating / evaluating the method as a stochastic process.  The experiments are useful as a proof of concept and the genotype imputation experiment presents a scenario where this model seems to make sense over standard deep learning approaches.  The UCI experiments are a bit weak in terms of the baselines and the overall results.  Overall I think the novelty and presentation warrant an accept but the paper could be stronger.  I would vote accept but I wouldn’t champion the paper strongly (e.g. for a presentation).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces semi-parametric inducing point networks, which learn a latent representation of the entire training set which can be queried at inference time efficiently. The authors experiment on the UCI benchmarks, a synthetic meta-learning task, and a genomic imputation task. The key contribution is a decrease in computational cost vs existing methods.
strength_and_weaknesses: Strengths:

This paper is excellently written. I enjoyed reading it - particularly the intro and background, and it flows nicely.

Overall, the results are slightly better than existing methods. But SPIN requires significantly less GPU usage and param count than existing methods, as well as allows for larger context size. Computational cost grows linearly in the training set size compared to  the quadratic growth of existing methods.

The experiments seem convincing, and I particularly like the genomic imputation experiments. This type of model seems like a good method for genomics, where we always have the reference genome and prior lab-experiments.


Weaknesses:

The motivation for why we need more GPU efficient methods compared to NPT isn't very clear. Table 2 shows the effect of a larger context size that NPT can't fit in memory, but aside from that I didn't see anything else. For something like genomic imputations, those aren't time-sensitive so it shouldn't be big deal waiting a little longer or using a little more compute.

The paper has a lot of equations, but it boils down to a fairly simple method using multi-head cross-attention. I appreciated the detail, but I think the paper could be greatly simplified.

The related work section lists a lot of papers, but doesn't provide much of a comparison to the proposed method.

What is the intuition of "folding" in XABD? I think this was a key part of the paper, but it wasn't motivated. Similarly, what is the intuition for ABLA?

Minor: 

What are "attributes" in XABA? I found this nomenclature confusing.

Figures should be vectorized.

I would move the "Attention Mechanisms" subsection to an appendix (aside from explaining what MAB is).

Tables and Figures could be organized a little bit better. Hard to read in the text when it's flowing in every direction around the figures and tables.

clarity,_quality,_novelty_and_reproducibility: The paper is very well written overall, aside from some explanation of the design choices and intuitions.

It's hard for me to speak on novelty since I'm not very familiar with the background work. Overall, it gave me impressions of the perceiver model (mentioned in related work), but applied to encode the entire training set.

The experiments seem nice, but again I don't know the standard procedure for semi-parametric model evaluations.

Code and data availability section in the Appendix is nice.
summary_of_the_review: Overall, I think this is a solid paper and it encourages future work on compressing training sets into queryable encodings. The key contribution seems to be the computational efficiency of their method. It's clear that it's more computational efficient at the same performance, but it's not very clear how important computational efficiency is here.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a mechanism for semi-parametric prediction using neural attention mechanisms and a variant of inducing points.

In this work, the authors combine ideas such as cross-attention between attributes and between datapoints, yielding a framework -denoted SPIN- which given a new dataset produces a set of lower dimensional inducing points H in an encoder structure.
This set H, can either be used to update to the next set H given data, or it can be used in a prediction layer in a comnstruct which allows performing predictions of the style p(y|x, H).

The authors propose to then utilize SPIN for a neural process variant, which using these inducing points is able to incorporate large context sizes in context datasets D_c to make predictions on query datasets.

In their experiments, they show competitive performance on various tasks related to the neural process literature where they exhibit strong scalability due to better memory utilization of their method compared to useful baselines.
Of particular note is an example for genotype imputation, which seems to benefit strongly in this meta-learning scenario from the proposed methodology.
strength_and_weaknesses: Strengths:
The technique the authors propose appears to work well empirically.
While the core setup is a familiar one of neural processes, the particular handling of inducing points here allows the models to remain competitive and scale beyond the typical comparators, which is a promising avenue for its future use as one may want to incorporate larger context datasets D_c.

The authors do a good job comparing to a few benchmarks such as UCI datasets, GP-prior modeling, and genotype imputation.

Weaknesses:
Some brief ablation experiments seem to indicate that the lion;s share of the work is carried by the cross-data attention modeling part here. It would be great to get deeper insight into this.
In general the work appears to propose an architecture which feels somewhat "engineered" and while the authors show that it works, it would be great to understand better why each part works, how sequencing these different layers makes sense , what calculation the model is really amortizing over here, and deepen the analysis of the ablations.
What does the ABLA layer really do? I understand the authors claim it refines H_A, but it is not clear to me what this is amortizing and the work does not really convey the intuition here.

clarity,_quality,_novelty_and_reproducibility: The authors write relatively clearly here and the work is empirically of high quality.

The writing is also of good quality, and the authors put great attention to detail into listing lots of the many details in their architecture, but by its own nature this work feels somewhat engineered and is not easy to deeply understand from a mathematical principles point of view, but rather as a list of specific implementations that has a desired effect. This is partly the nature of the work, but I would hope for more understanding why these techniques perform well.
What the authors do a great job on is explaining the memory and computational footprint of this architecture, which is also a key aspect of its ability to scale. I came away with a good understanding of why this would scale well on larger context sets thanks to the writing.

I was appreciative of the authors' great job to position this work fairly among newer and older literature, the scholarship here is thorough and fair and spans all walks of ML where inducing points and meta-learning/neural processes have been seen.

With respect to reproducibility, I do believe I could reimplement the key layers here after reading the paper and the authors share lots of detail in their appendix. By its nature -again- work like this probably depends strongly on the details of the experimental tuning, I came away with the impression the authors are handling this well.
summary_of_the_review: The authors present an evolution and marriage of the streams of work on neural processes and cross-datapoint attention, which they bring together to propose an architecture which can utilize large context sets to perform semi-parametric prediction.
I really enjoyed the key application to genotype imputation as an example that would be uniquely enabled by this model.

Overall the paper is of good enough quality and interest and although it does not meaningfully extend or propose much technical novelty or deep insights into why the utilized pieces here work as well as they do, the empirical qualities regarding scalability and positioning of the paper appear and useful and valuable.

If the authors were able to better explain and understand the interplay of their layers and which computations they are amortizing I would have been more excited.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new semi-parametric learning method called SPIN. It builds upon non-parametric transformers (NPT), which use attention between training points to learn compact and effective predictors at the cost of quadratic runtime in the number of training points considered (context size). SPIN addresses this by utilizing inducing points, a small constant number of vectors summarizing the large number of training points. The resulting architecture scales linearly in the context size at training time, and linearly only in the selected number of inducing points at test time. It is demonstrated that depending on the selected context size, SPIN yields improved prediction performance, reduced computational cost, or both. Additionally, SPIN is employed for metalearning on synthetic and genome imputation data, where it delivers improved/competitive performance, respectively.
strength_and_weaknesses: Strengths:
 - The paper is well motivated, as it addresses a clear weakness of NPT in a suitable and convincing manner.
 - The method is clearly explained and sensibly constructed.
 - The experimental evaluation is thorough and highlights important aspects of the method, including its sensitivity to context size, its parameter count, and its memory usage.
 - The improvements over NPT on UCI and meta learning experiments are quite convincing.

Weaknesses:
 - On the genome imputation experiment, there is no significant performance increase over prior methods, the result is only notable in terms of SPIN's generality and low parameter count. Maybe some more concretely desirable property such as inference time could be measured here for a more convincing argument.
 - The resolution of the figures, especially figure 3, should be improved, ideally by making them a vector graphics.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow. 

The experiments are thorough and high quality. 

Using inducing points in the context of NPT is novel to my knowledge, and the authors have done a good job building upon this idea by including large scale meta-learning experiments.

Reproducibility of the method is ensured by a thorough appendix and provided code.

Question for the authors: How do you explain the performance gains in Table 2 over NPT when the context size is identical? Intuitively, it seems like the bottleneck introduced by inducing points should adversely impact performance.
summary_of_the_review: In summary, I think the paper represents a clear step forward for semi-parametric methods. The idea of using inducing points is well executed and thoroughly evaluated. The meta-learning and genome imputation experiments push the applicability of such methods to new domains. I therefore recommend acceptance.

-----

Post Rebuttal Update: After reading the other reviews and the authors' responses, I maintain my positive view of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes semi-parametric inducing point networks (SPIN) as an instance of neural processes. The major innovation here is the design of inducing points and attention blocks to achieve linear run-time cost as the number of datapoint grows.

Reviewers found that the paper is well written, the genotype imputation application is interesting, and the computational gains of SPIN over NPT could be useful in certain practical scenarios.

However, the SPIN approach doesn't seem to provide consistent performance gains over the baseline. 

Overall, I think the improved attention architecture with computational gains is useful for the field of neural processes and set data modelling, and I believe researchers in this field can gain some insights from reading this paper.

As a side note: as AC I cannot see the authors' replies to the reviewers (perhaps because the Reader fields were incorrectly specified), so I reached out to ask the authors' reply regarding improvements over the previous submission version of this paper.
justification_for_why_not_higher_score: Computational gain is significant, however the proposed SPIN approach does not robustly out-perform NPT which is the paper's main baseline.
justification_for_why_not_lower_score: Reviewers all recommend accept in their review. The computational gain of SPIN over NPT is worth presenting at the conference.
Decision: Accept: poster