Semi-Parametric Inducing Point Networks and Neural Processes
Authors: Richa Rastogi
Yair Schiff
Alon Hacohen
Zhaozhi Li
Ian Lee
Yuntian Deng
Mert R. Sabuncu
Volodymyr Kuleshov
Venue: ICLR 2023 poster (2023)

Abstract: We introduce semi-parametric inducing point networks (SPIN), a general-purpose architecture that can query the training set at inference time in a compute-efficient manner. Semi-parametric architectures are typically more compact than parametric models, but their computational complexity is often quadratic. In contrast, SPIN attains linear complexity via a cross-attention mechanism between datapoints inspired by inducing point methods. Querying large training sets can be particularly useful in meta-learning, as it unlocks additional training signal, but often exceeds the scaling limits of existing models. We use SPIN as the basis of the Inducing Point Neural Process, a probabilistic model which supports large contexts in meta-learning and achieves high accuracy where existing models fail. In our experiments, SPIN reduces memory requirements, improves accuracy across a range of meta-learning tasks, and improves state-of-the-art performance on an important practical problem, genotype imputation.

Published as a conference paper at ICLR 2023
INDUCING POINT NETWORKS AND NEURAL PROCESSES
Richa Rastogi; Yair Schiff, Zhaozhi Li, Ian Lee, Mert R. Sabuneu, & Volodymyr Kuleshov'
Cornell University
(rr568, yz.2,, z 643,, y1759, msabuncu, kuleshov) ( @cornell.eddeeeeeoeeeeeeeee
Alon Hacohen
Yuntian Deng
Technion Israel Institute of Technology Alonhacchen@campss technion. ac il
Harvard University
dengyuntian@seas. hww.:eteieeeeeaaa.aa....ee. harvard.
ABSTRACT
We introduce cami-paraeetric inducing point networks (SPIN), general-purpose architecture that can query the training set at inference time in a compute-efficient manner. Semi- parameerattr architectures are typically more compact than parametric models, but their computational complexity is often quadratic. In contrast, SPIN attains linear complexity via cross-attention mechanism between datapoints inspired by inducing point methods. Querying large training sets can be particularly useful in meta-leanee leatalle as it unlocks additional training signal, but often exceeds the scaling limits of existing models. We use SPIN as the basis of the Inducing Point Neural Process, a probabilistic model which supports large contexts in meta- learning and achieves high accuracy where existing models fail. In experiments, SPIN reduces memory requirements, improves accuracy across a range of meta- learning tasks, and improves state-of-the-art performance important practical problem, genotype imputation.
INTRODUCTION
Recent advances in deep learning have been driven by large-scale parametric models (Krizhevsky et al., 2012; Peters et al., 2018; Devlin et al., 2019; Brown et al., 2020; Ramesh et al., 2022). Modern parametric models rely on large numbers of weights to capture the signal contained in the training set and to facilitate generalization (Frankle & Carbin, 2018; Kaplan et al., 2020); as a result, they require non-trivial computational resources (Hoffmann et al., 2022), have limited interpretability (Belinkov, 2022), and impose a significant carbon footprint (Bender et al., 2021).
This paper focuses on alternative semi-parametric approach, in which we have access the training at inference time and learn parametric mapping y fo Dirain) conditioned on this dataset. demi-paramettis models can query the training set Dirain and can therefore express rich and interpretable mappings with compact fo- Examples of the semi-parametria framework include retrieval- retrieval-augmented language models (Grave et al., 2016; Guu et al., 2020; Rae al., 2021) and non-parametric transformers (Wiseman & Stratos, 2019; Kossen et al., 2021). However, existing approaches are often specialized to specific tasks (e.g.. language modeling (Grave al., 2016; Guu al., 2020; Rae et al., 2021) or sequence generation (Graves et al., 2014)), and their computation scales superlinearly in the size of the training set (Kossen et al., 2021).
set Durain (x(*) y ( Je=1
Here, we introduce cami-parametric inducing point networks (SPIN), general- purppse architecture whose computational complexity at training time scales linearly in the size of the training set Dirain and in the dimensionality of and that constant in Deruin at inference time. Our architecture is inspired by inducing point approximations (Snelson & Ghahramani, 2005; Titsias, 2009; Wilson & Nickisch, 2015; Evans & Nair, 2018; Lee al., 2018) and relies on cross-attention mechanism between datapoints (Kossen et al., 2021). An important application of SPIN is in meta-learning, where conditioning on large training sets provides the model additional signal and improves accuracy,
"Correspondence Richa Rastogi and Volodymyr Kuleshov

---

Published as a conference paper at ICLR 2023
but poses challenges for methods that scale superlinearly with Drrain- We use SPIN the basis of the Inducing Point Neural Process (IPNP), a scalable probabilistic model that supports accurate meta-learning with large context sizes that cause existing methods to fail. We evaluate SPIN and IPNP on a range of supervised and meta-learning benchmarks and demonstrate the efficacy of SPIN on a real-world task in genomics genotype imputation (Li et al., 2009). In meta-learning experiments, IPNP supports querying larger training sets, which yields high accuracy in settings where existing methods run out of memory. In the genomics setting, SPIN outperforms highly engineered state- arr software packages widely used within commercial genomics pipelines (Browning et al., 201 8b), indicating that our technique has the potential to impact real-worle systems.
Contributions In summary, introduce SPIN, pamimiaraettric neural architecture inspired by inducing point methods that is the first to achieve the following characteristies:
1. Linear time and space complexity in the size and the dimension of the data during training.
2. The ability to learn compact encoding of the training set for downstream applications. As a result, at inference time, computational complexity does not depend on training set size.
We use SPIN as the basis of the IPNP, a probabilistic model that enables performing meta-learning with context sizes that are larger than what existing methods support and that achieves high accuracy on important real-world tasks such as genotype imputation.
BACKGROUND
Parametric and Semi-Parametrie Machine Learning Most supervised methods in deep learning parametric. Formally, given training set Dtrain = (x((0) y(0 Ji=1 with features X and labels y E y, we seek to learn a fixed number of parameters E O of a mapping y = fo(x) using supervised learning. In contrast, non-parametric approaches learn mapping y fo(x Derain) that can query the training set Derain at inference time; when the mapping fe has parameters, the approach is called peri-parametric. Many deep learning algoritmss-includnng memory-augment architectures (Graves et al., 2014; Santoro et al., 2016), retrieval- -based language models (Grave et al., 2016; Guu et al., 2020; Rae al., 2021), and non parametrie transformers (Kossen al., 2021) are instances of this approach, but they are often specialized to specific tasks, and their computation scales superlinearly in n. This paper develops scalable and domain-agnostic semi- aani-parametric methods.
Meta-Learning and Neural Processes An important application of semi comi-parametric methods is in meta- meta-learning, where we train a model to achieve high performance on new tasks using only a small amount of data from these tasks. Formally, consider a collection of D datasets (or meta-
dataset) (D(d) )D 1, each defining a task. Each D(d) = (D( D( contains a set of context points .y:
D! = {xx")) (di) (di) mm and target points D( {x} (di) (di) 1m 1=-1- Meta-learning seeks produce
-ye
a model fx; De) that outputs accurate predictions for on Dr and on pairs (De, D.) not seen training time. Neural Process (NP) architectures perform uncertainty aware meta-learning by mapping context sets to representations re (D))),, which can be combined with target inputs to provide distribution on target labels y: p(y [x,, (D.)), where p a probabilistic model. Recent successes in NPs have been driven by attention based architectures (Kim et al., 2018; Nguyen & Grover, 2022), whose complexity scales super-lineatte In concurrent work, Feng et al. (2023) propose a linear time method, using cross attention to reduce the size of context datasets.
with context size D our method yields linear complexity.
A Motivating Application: Genotype Imputation A specific motivating example for developing efficient semi-parametric methods the problem of genotype imputation. Consider the problem of determining the genomic sequence y E (A, C, G. of individual; rather than directly measuring y, it is common to use an inexpensive microarray device to measure a small subset of genomic positions E (A, T, C, G)P, where p < k. Genotype imputation is the task of determining y from x via statistical methods and a dataset Ptrain {x()) (0) in Ji=1 of fully-sequenced individuals (Li et al., 2009). Imputation is part of most standard genome analysis workflows. It is also a natural candidate for semi- camipaaraettric approaches (Li & Stephens, 2003): a query genome y can normally be represented as combination of sequences (() from Drrain because of the biological principle of
2

---

Published as a conference paper at ICLR 2023
recombination (Kendrew, 2009), shown in Figure Additionally, the problem is poor fit for parametric models: can be as high as 10 and there is little correlation across non- aonoriiiial parts of y. Thus, we need an unwieldy number of parametric models (one per subset of y), whereas single semi-parametric model can run imputation across the genome.
Attention Mechanisms Our approach for designing semi parammmratttric models relies modern attention mechanisms (Vaswani et al., 2017), specifically dot-product atlention (Ll(Q, K, V), which combines query matrix Q E Rdg Kea with key and value matrices K Rdd V Rdd Xev as
Subject's
Genome
Unrelated Individuals
Att(Q.K, = softmax (QK /QK///// /V
Most genomes can described as mosalc made from small number of blocks that be learned from data
To attend to different aspects of the keys and values, multi-heas head attention (MHA) extends this mechanism via е₁ attention heads:
Figure 1: Genotype recombination
MHA(Q. K, V) concat( O1....Oe,).
O, Atw Kw VW))
Each attention head projects Q, K, V into lower-dimensional space using learnable projection matrices Wº w Re wy Rev: and mixes the outputs of the heads using W Reaecs Xeo As is commonly done, we assume that Cok Ca/en, Eigh eg/en, and C. Cg. Given two matrices X, H Raxe. mulli-head attention block (MAB) wraps MHA together with layer normalization and a fully connected layer':
MAB(X, H) O FF(LayerNorm(O) (O))
O - X + MHA (LayenNorm (X)) H, H)
Attention parimpaaaettri models normally scales quadratically in the dataset size (Kossen et al., 2021); our work is inspired by efficient attention architectures (Lee et al., 2018; Jaegle et al., 2021b) and develops scalable cmi-pparamttric models with linear computational complexity.
SEMI-PARAMETRIC INDUCING POINT NETWORKS
3.1 SEMI-PARAMETRIE LEARNING BASED ON NEURAL INDUCING POINTS
A key challenge posed by semi-parametric methods one (Hearst et al., 1998) as well recent dienttio-bbsee approaches (Kossen et al., 2021)-is the On?2) computational cost per gradient update at training time, due to pairwise comparisons between training set points. Our work introduces methods that reduce this cost to Ohhn) where < is rrper-parameter-.e prrerrpaaaeter-withooe without sacrificing performance.
one affecting both classical kernel methods
Neural Inducing Points Our approach is based on inducing points, technique popular in ap- proximate kernel methods (Wilson Nickisch, 2015; Lee et al., 2018). set of inducing points H (h6))A (1)))_1 can be thought as a "virtual" set of training instances that can replace the training set Durain. Intuitively, when Dirain is large, many datapoints are redundant example, groups of similar ( can be replaced with a single inducing point (() with little loss of information.
The key challenge developing inducing point methods is finding a good set H. While classical approaches rely on optimization techniques (Wilson & Nickisch, 2015), we use an attention mech- anism to produce H. Each inducing point h() E H attends over the training set Dtrain to select its relevant "neighbors" and updates itself based them. We implement attention between and Dirain in (hh) time.
Dataset Encoding Note that once we have good set of inducing points H. becomes possible to discard Dtrain and use H instead for all future predictions. The parametric part of the model makes predictions based on H only. This feature is an important capability our architecture; computational complexity is now independent of D and we envision this feature being useful in applications where sharing D Dirain not possible (e.g., for computational or privacy reasons).
We use the pre-norm parameterization for residual connections and omit details such as dropout, see Nguyen Salazar (2019) for the full parameterization.
3

---

Published as a conference paper at ICLR 2023
3.2 PAMI-PARAMETRIC INDUCING POINT NETWORKS
Next, we describe semi-paramens inducing point networks (SPIN), domain-agnostic architecture with linear-time complexity.
Notation and Data Embedding The SPIN model relies training set Dtrain fxx(), . y
with imput features x()) X and labels yr() Y where X, y V, which is the input and output vocabulary 2. We embed each dimension (each attribute) of x and into an e-dimensional embedding and represent Derain tensor of embeddings D Embed D Rxxxxxe, where d = p is obtained from concatenating the sequence of embeddings for each x(1) and y(
The set Dtrain is used to learn inducing points H { similarly, represent H via tensor HC fxe of S inducing points, each being a sequence of f I d embeddings of size e.
To make predictions and measure loss set of b examples D query {x("),
the same embedding procedure obtain tensor of input embeddings X querry Rxxxxe by embedding (x(²), - in which the labels have been masked with zeros. We also to store the ground truth labels and inputs (the objective function we use requires the model to make predictions on masked input elements as well, see below for details).
, y
1b JI=1.
tensor Y gold R
High-Level Model Structure Figure 2
presents an overview of SPIN. At a high level, there are two components: (1) an Encoder module, which takes as input Derain and returns tensor of inducing points H; and (2) Predictor module, which is fully parametric model that outputs logits Y query from H and querr--
D
Encoder
H
Xquay
Predictor
Y query
Figure SPIN Model Structure
D = (mbrai(trrin)
H Encoder(D)
Y query PrredictorXqqueet H)
The encoder consists of sequence of layers, each of which takes as input E RRxxxx and two tensors HA E =x and HD E fxx and output updated versions of HA, HD for the next layer. Each layer consists of a sequence of up to three cross-attention layers described below. The final output H of the encoder is the Hp produced by the last layer.
3.2.1 ARCHITECTURE OF THE ENCODER AND PREDICTOR
Each layer of the encoder consists of three sublayers denoted as XABA, XABD, ABLA. An encoder layer takes as input HA, HD and feeds its outputs H',, H', (defined below) into the next layer. The initial inputs HA, Hp of the first encoder layer are randomly initialized learnable parameters.
H' XABA(HA,
H XABD(H,. H',)
HA* ABLA(H'))
Cross-Attention Between Attributes (XABA) An XABA layer captures the dependencies among attributes via cross-attention between the sequence of latent encodings in H and the sequence of datapoint features in D.
XABA(HA, D) MAB(HA, D)
This updates the features of each datapoint in HA to be combination of the features of the corresponding datapoints in D. In effect, this reduces the dimensionality of the datapoints (from d e to n f x e). The time complexity of this layer is O(ndfe), where the dimensionality of the reduced tensor.
Cross-Attention Between Datapoints (XABD) The XABD layer the key module that takes into account the entire training set to generate inducing points.
First, reshapes ("unfoods"") its input tensors H' E Raxffx and HD E Axxfxe into ones of dimensions (1 x x fe) and (I x h fe) respectively. It then performs cross-attention between
Heee consider the case where both input and output discrete, but our approach casily generalizes to continuous input and output spaces.
4

---

Published as a conference paper at ICLR 2023
XABA
ABLA
XABD
Induced
Dataset Encoder
Latent Ho (1. te)
Attribute Latent HA (n,
HAY
x N Layers
Optional
Cross Attention
Cross Attention
Predictor Module
Training Dataset (n,
Xquery
query
Qquery
Figure 3: SPIN Architecture. Each layer of the encoder consists sublayers XABA, ABLA and XABD, and the predictor consists of cross attention layer. We omit feedforward layers for simplicity.
the two unfolded tensors. The output of tross-attentie has dimension (1 x h x fe); it is reshaped ("folded") into an output tensor of size (h x f x e).
XABD(HD. H¹,) = fold [MA((unfoldHH). unfold(H',))
This layer produces inducing points. Each inducing point HD attends to innnensionnaity-rcuuce datapoints in H', and uses its selected datapoints to update its own representation. The computational complexity of this operation (nn which is linear training set size n.
Self-Attention Between Latent Attributes (ABLA) The third type of layer further captures dependencies among attributes by computing regular self-attention across attributes:
ABLA(HA)) = MAB(H', H',)
This enables the inducing points to refine their internal representations. The dataset encoder consists of a sequence of the above layers, see Figure The A ABLA layers are optional based on validation performance. The input HD to the first layer is part of the learned model parameters; the initial HA is a linear projection of D. The output of the encoder is the output HDD of the final layer.
Predictor Architecture The predictor is parametric model that maps input tensor X Xquery to an output tensor of logits Y querry. The predictor can use any parametric model. We propose architecture based simple cross-attention operation followed by linear projection to the vocabulary size, as shown in Figure 3:
Predict(X (X query H) (F(MAA((nfod((xpueer) unfold(H))
queryy:
3.3 INDUCING POINT NEURAL PROCESSES
An important application of SPIN is in meta-learning, where conditioning on larger training sets provides more information the model, and therefore has potential improve predictive accu- racy. However, existing methods scale superlinearly with Drrains and may not effectively leverage large contexts. We use SPIN the basis of the Inducing Point Neural Process (IPNP), scalable probabilistic model that supports fast and accurate meta-lcarning on large context sizes.
IPNP detines probabilistic model p(y xx r(x, D.)) of a target variable y conditioned on an input x and a context dataset D... This context is represented via fixed-dimensional context vector r(x, D.), and we use the SPIN architecture to parameterize where Encoder is the SPIN encoder, producing a tensor of
function of De. Specifically,
define Fe = (embeedr(Emed(D.... inducing points. Then, we compute r(x, re) MAB(x, is a distribution with parameters Φ(x, r), e.g., Normal distribution with á (n, Σ) or Bernoulli with Φ E [0, 1]. We parameterize the mapping (xx,r) with fully-connected neural network.
cross- attention. The model p(y)x (
We further extend IPNPs to incorporate latent variable z that is drawn from Gaussian p(z D.) parameterized by Φ₂ m points. This latent variable can be thought of as capturing global uncertainty (Garnelo et al., 201 8). This results a distribution p(y 2 2/x, Dz) = p(y|z x, (z)))(z(D.), where p(y z, x, Dz) is parame- terized by o(z, x. re), with ó itself being fully connected neural network. See Appendix A.6 for more detailed architectural breakdowns. Following terminology the NP literature, we refer to our model conditional IPNP (CIPNP) when there is latent variable z present.
(D.)), where m represents mean pooling across data-

---

Published as a conference paper at ICLR 2023
3.4 OBJECTIVE FUNCTION
SPIN We train SPIN models using a supervised learning loss Llavels (e.g., E2 loss for regression, cross-entropy for classification). We also randomly mask attributes and add additional loss term Latuibutes that asks the model to reconstruct the missing attributes, yielding the following objective:
SPIN = (1 - X) claess + A.Catriibnes
Following Kossen et al. (2021), we start with weight À of 0.5 and anneal it to lean towards zero. We detail the loss terms and construction mask matrices over labels and attrbutes Appendix A.2. Following prior works (Devlin et al., 2019; Ghazvininejad et al., 2019; Kossen et al., 2021), we use random token level masking. Additionally, we propose chunk masking, similar to the span masking introduced in (Joshi al., 2019), where fraction p of the samples selected have the mask matrix for labels ( 1, and we show the effectiveness of chunk masking in Table 5.
IPNP Following the NP literature, IPNPs are trained on meta-dataset (D((d) )a= of context and training points D(d) = (D! D( to maximize the log likelihood of the target labels under the learned parametric distribution LIPNP b Σx 1 log
(di) D! x; ((()). For latent
variable NPs, the objective is variational lower bound; see Appendix A.6 for more details.
4 EXPERIMENTS
Semi Semi-parametric models-includiee Neural Processes for meta-learning benefft from large context sets D., as they provide additional training signal. However, existing methods scale superlinearly with D. and quickly run out of memory. In experiments section, we show that SPIN and IPNP outperform state-of of-t-e--r-t models by scaling to large De that existing methods do not support.
4.1 UCI DATASETS
We present experimental results for 10 standard UCI benchmarks, namely Yacht, Conerete, Boston- Housing, Protein (regression datasets), Kick, Income, Breast Cancer, Forrest Cover, Poker-Harl and Higgs Boson (classification datasets). We compare SPIN with Transformer baselines such as NPT (Kossen et al., 2021) and Set Transformers (Set-TF) (Lee et al., 2018). We also evaluate against Gradient Boosting (GBT) Friedman (2001), Multi Layer Perceptron (MLP) (Hinton, 1989; Glorot Bengio, 2010), and K-Nearest Neighbours (KNN) (Altman, 1992).
Table 1: Performance Summary on UCI Datasets
Traditional ML
Transformer
GBT
MLP
KNN
NPT
Set-TF
SPIN
Ranking 4 GPU Mem +
3.00
4.10+1.77
5.44+1.01
2.30
3.63+00.2
2.10-0088
1.0x
1.39-0067x
0.46++0.21x
Following Kossen et al. (2021), we measure the average ranking of the methods and standardize across all UCI tasks. To show the memory ef- ficiency of our approach, we also re- port GPU memory usage peaks and as a fraction of GPU memory used by NPT for different splits of the test dataset in Table 1.
Table Effect of context size on the Poker Hand dataset
Context Size 10K
Approach
4096
15K
30K
NPT Acct
80.11
OOM
Mem.
9.82
0OM
SPIN Acct
82.98
95.99
96.06
99.43
Mem
1.73
3.88
5.68
10.98
GBT
MLP
KNN
Results SPIN achieves the best
average ranking 10 UCI datasets and uses half the GPU memory
Acct 71.88 1.5.91
66.09
+9.88
54.75 +-0.03

---

Published as a conference paper at ICLR 2023
compared to NPT. We provide detailed results each of the datasets and hyperparameter details in Appendix A.4. importantly, SPIN achieves high performance by supporting larger context sets-we illustrate this in Table 2, where compare SPIN and NPT on the Poker Hand dataset (70/20/10 split) using various context sizes. SPIN and NPT achieve 80-82% accuracy with small contexts, but the performance of SPIN approaches 99% context size is increased, whereas NPT quickly runs out of GPU memory and fails to reach comparable performance.
4.2 NEURAL PROCESSES FOR META-LEARNING
Experimental Setup Following previous work (Kim et al., 2018; Nguyen & Grover, 2022),
we perform a Gaussian process meta-learning experiment, for which we create collection of datasets (D()) )2= where each D(4) contains random points (x("))² where x(4) I, and target points y y( f(a) (x(d)) obtained from function f(d) sampled from Gaussian Pro- cess. At each meta-training step, we sample B = 16 functions {f 10=1- For each f(h, max_ctx] context points and n. ~ U [min_ttt, max_tgt] target points. The range for is fixed across all experiments at [4, 64). The range for is varied from [64, 128]. [128, 256], [256, 512], [512, 1024], [1024, 2048]. We train several different NP models for 100,000 steps and evaluate their log- likelihood on 3,000 hold out batches, with B, m, n taking the same values as at training time. We evaluate conditional (CIPNP) and latent variable (IPNP) variations of our model (using h 2 mmin_etxx ctx inducing points) and compare them to other attention- NPs: Conditional ANPs (CANP) (Kim et al., 2018), Bootstrap ANPs (BANP) (Lee et al., 2020), and latent variable ANPs (ANP) (Kim et al., 2018).
sample m ~ [[min__ctt
Results The IPNP models attain higher performance than all baselines most context sizes (Figure 4). Interestingly, IPNPs generalize better-recall that IPNPs are more compact models with fewer parameters, hence are less likely to overfit. We also found that increased context size led to improved performance for all models; however, baseline NPs required excessive resources, and BANPs ran out of memory entirely. In contrast, IPNPs scaled to large context sizes using up to 50% less resources.
1.35
Eval LL
1.30
CANP
CIPNP
BANP
ANP
00M
Avg step time (ms)
80
60
20
8] 88. 9] 10] [10, 11] Ctx. points [logz(min__ctx))
16.
4.3 GENOTYPE IMPUTATION
Genotype imputation the task of inferring the sequence y of an entire genome via statistical methods from a small subset of positions x-usually obtained
Figure 4: Inducing Point NPs outperform NP baselines and train much faster. Plots display mean + std. deviation from 5 runs with different random seeds.
from an inexpensive DNA microarray device (Li et al.,
1==1 2009>-and a dataset Drrain (x()), (0) of fully- sequenced individuals (Li et al., 2009). Imputation is part of most standard workflows in genomics (Lou et al., 2021) and involves mature imputation software (Browning et al., 201 Rubinacci et al., 2020) that benefits from over a decade of engineering (Li & Stephens, 2003). These systems are fully non-parametric and match genomes in Dirain to y.y; their scalability to modern datasets of up to millions of individuals a known problem in the field (Maarala et al., 2020). Improved imputation holds the potential reduce sequencing costs and improve workflows in medicine and agriculture.
Experiment Setup We compare against one of the state-of-the-arr packages, Beagle (Browning et al., 2018a), on the 1000 Genomes dataset (Clarke et al., 2016), following the methodology described in Rubinacei et al. (2020). We use 5008 complete sequences y that we divide into train/val/test splits of 12/0.02, respectively, following Browning et al. (2018) 8b). We construct inputs x by masking positions that do not appear on the Illumina Omni2.5 array (Wrayner). Our experiments in Table focus on five sections of the genome for chromosome 20. We pre-process the input into sequences of K K-mers for all methods (see Appendix A.3). The performance of this task is measured via the Pearson correlation coefficient R² between the imputed SNPs and their true value at each position.
7

---

Published as a conference paper at ICLR 2023
We compare against NPTs, Set Transformers, and classical machine learning methods. NPT- 16, SPIN-16 and Set Transformer- 16 refer to models using an embedding dimension 16, a model depth of 4, and one attention head. NPT-64, SPIN-64 and Set Transformer-64 refer to models using an embedding dimension of 64, a model depth of 4, and 4 attention heads. SPIN uses 10 inducing points for datapoints (h=10, f=10). A batch size of 256 is used for Transformer methods, and train using the lookahead Lamb optimizer (Zhang et al., 2019).
Table Performance Summary on Genomic Sequence Imputation. (+) represents parametric models. A difference of 0.5% is statistically significant at pvalue 0.05.
GBT
MLP
KNN
Beagle
NPT-16
Set-TF-16
SPIN-16
Pearson R2 t 87.63
95.31
89.70
95.64
95.84 =0.06
95.97=0.09
95.92 #0.12
Param Count
65M
16.7M
33.4M
8.1M
Results Table 3 presents the main results for genotype imputation. Compared to the previous state-of-the-art commercial software, Beagle, which is specialized to this task, all Transformer- based methods achieve strong performance, despite making fewer assumptions and being more general. While all the three Transformer-basee approaches report similar Pearson R², SPIN achieves competitive performance with much smaller parameter count. Among traditional ML approaches, MLP perform the best, but requires training one model per imputed SNP, and hence cannot scale to full genomes. We provide additional details on resource usage and hyper-parameter tuning in Appendix A.3.
4.4 SCALING GENOTYPE IMPUTATION VIA META-LFARNIH
One of the key challenges in genotype imputation is making predictions for large numbers of SNPs. To scale to larger sets of SNPs, we apply a meta-learning based approach, in which a single shared model used to impute arbitrary genomic regions.
Experimental Setup We create a meta-training set (D(4)) where each (D! D( corre- sponds to of D independent genomic segments, D( is the set of reference genomes in that segment, and (. is the set of genomes that we want to learn to impute. At each meta-training step, we sample new pair (D(d) D(!)) and update the model parameters to maximize the likelihood of Df We further create three independent versions this experiment--dentee Full, 50%, and 25%-in which the segments detining (D!d), DD(d)) contain 400, 200, and 100 SNPs respectively. We fit an NPT and a CIPNP model parameterized by SPIN-64 architecture and apply chunk-level masking method instead of token-level masking.
Table 4: Multiple Windows Experiment
Results Table shows that both the CIPNP (SPIN-64) and the NPT-64 model support the meta-learning approach to genotype imputation and achieve high performance, with CIPNP being more accurate. We provide performance for each region within the datasets in Appendix A.3, Table 8. How- ever, the NPT model cannot handle full-length ge- nomic segments and runs out of memory the full experiment. This again highlights the ability of SPIN to scale and thus solve problems that existing models cannot.
25%
50%
Full
NPT-64 R² 1
95.06
92.89
0OM
Mem +
12.36
19.86
00M
SPIN-64 R² 1
95.38
93.55
93.90
Mem 5.33
8.30
16.44
Table 5: Masking
Masking Table 5 shows the effect of chunk style masking over token level masking for SPIN in order to learn the imputation algorithm. As the genomes are created by copying over chunks due to the biological priniciple of recombination, we find that chunk style masking labels at train time provides significant improvements over random token level masking for the meta learning genotype imputation task.
Masking R²²
Token Chunk 95.32
80.48
8

---

Published as a conference paper at ICLR 2023
Ablation Analysis To evaluate the effectiveness of each module, we perform ablation analysis by gradually removing components from SPIN. We remove components one at time and compare the performance with default SPIN configuration. In Table 6, observe that for the genomics dataset (SNPs 424600-424700) and UCI Boston Housing (BH) dataset, both XABD and XABA crucial components. We discuss ablation with synthetic experiment setup in Appendix A.7
Table Ablation Analysis
GEN
BH 4
SPIN
94.05
3.0 10.6
-XADD
93.50
3.1 #0.8
-XABA
93.89
3.2 +1.5
5 RELATED WORK
Von-Parametric and based on kernels (Davis et al., 2011), such as Gaussian processes (Rasmussen, 2003) and support vector machines (Hearst et al., 1998). These methods feature quadratic complexity (Bach, 2013), which motivates a long line of approximate methods based on random projections (Achlioptas et al,, 2001), Fourier analysis (Rahimi & Recht, 2007), and inducing point methods (Wilson et al., 2015). Inducing points have been widely applied kernel machines (Nguyen et al., 2020), Gaussian processes classification (Izmailov & Kropotov, 2016), regression (Cao et al., 2013), semi-supervised learning (Delalleau et al., 2005), and more (Hensman et al., 2015; Tolstikhin et al., 2021).
Methods Non-parametric methods include approaches
Deep Semi-Parametric Models Deep Gaussian Processes (Damianou & Lawrence, 2013), Deep Kernel Learning (Wilson al., 2016), and Neural Processes (Garnelo et al., 2018) build upon classical methods. Deep GPs rely on sophisticated variational inference methods (Wang et al., 2016), making them challenging to implement. Retrieval augmented transformers (Bonetta et al., 2021) to query external datasets in specific domains such as language modeling (Grave et al., 2016), question answering (Yang al., 2018), and reinforcement learning (Goyal et al., 2022) and in way that is models (Graves et al., 2014). Non-Parametrie Transformers
attention
similar to earlier memory-augmentt (Kossen et al., 2021) use domain-agnostic architecture based attention that runs in O(n2d?) training time and (ndd) at inference time, while ours runs in ((nd) and (d)) respectively.
Attention Mechanisms The quadratic cost of self-attention (Vaswani et al., 2017) can be reduced using efficient architectures such as sparse attention (Beltagy et al., 2020). Set Transformers (Lee et al., 2018), the Performer (Choromanski et al., 2020), the Nystromer (Xiong et al., 2021), Long Ranger (Grigsby et al., 2021), Big Bird (Zaheer et al., 2020), Shared Workspace (Goyal et al., 2021) the Perceiver (Jaegle et al., 202 b;a), and others (Katharopoulos et al., 2020; Wang et al., 2020). Our work most closely resembles the Set Transformer (Lee al., 2018) and Perceiver (Jaegle et al., 202 10;));a) mechanisms-we extend these mechanisms to cross-attention between datapoints and use them to attend to datapoints, similar to non-maramt Transformers (Kossen al., 2021).
Set Transformers Lee et al. (2018) introduce inducing point attention (ISA) blocks, which replace self-attention with a more efficient cross-attention mechanism that maps a set of d tokens to new set of d tokens. In contrast, SPIN gross-attentiin compresses sets of size d into smaller sets of size h d. Each ISA block also uses a different set of inducing points, whereas SPIN layers iteratively update the same of inducing points, resulting in smaller memory footprint. Finally, while Set Transformers perform cross-attention over features, SPIN performs cross-attention between datapoints.
6 CONCLUSION
In this paper, we introduce a domain-agnosse general-purposs architecture, the semi-parametric inducing point network (SPIN) and use as the basis for Induced Point Neural Process (IPNPs). Unlike previous eemi-parametric approaches whose computational cost grows quadratically with the size of the dataset, our approach scales linearly the size and dimensionality leveraging a cross attention mechanism between datapoints and induced latents. This allows our method to scale to large datasets and enables meta learning with large contexts. We present empirical results 10 UCI datasets, a Gaussian process meta learning task, and real-world important task in genomics, genotype imputation, and show that our method can achieve competitive, if not better, performance relative to fate-of-the-ar methods at fraction of the computational cost.
the data by

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGMENTS
This work was supported by Tata Consulting Services, the Cornell Initiative for Digital Agriculture, the Hal & Inge Mareus PhD Fellowship, and an NSF CAREER grant (#2145577). We would like to thank Edgar Marroquin for help with preprocessing of raw genomic data. We would like to thank NPT authors Jannic and Neil for helpful discussions and correspondence regarding NPT architecture. We would also like to thank the anonymous reviewers for their significant effort to provide suggestions and helpful feedback, thereby improving our paper.
REPRODUCIBILITY
We provide details on the compute resources in Appendix A. including GPU specifications. Code and data used to reproduce experimental results are provided in Appendix C. We provide error bars on the reported results by varying seeds different test split, however for certain large datasets, such as UCI datasets for Kick, Forest Cover, Protein, Higgs and Genomic Imputation experiments with large output sizes, we reported results single run due computational limitations. These details are provided in Appendix A.3, Appendix A.4 and Appendix A.5.
REFERENCES
Dimitris Achlioptas, Frank McSherry, and Bernhard Schölkopf. Sampling techniques for kernel methods. Advances in neural information processing systems, 14, 2001.
N.S. S. Altman. An introduction to kernel and nearest-neighhon nonparametric regression. The American
Statistician, 46(3):175-185, 1992. doi: 10. 10.1080/000331005. 1992...599999..977799 URL https:// www. www.taannooiinini com doi/ abs 10. 1080/ 010/8000000000 000000003300 19..1155..99999.
Francis Bach. Sharp analysis of low-rank kernel matrix approximations. In Conference on Learning Theory, pp. 185 209. PMLR, 2013.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219, 2022.
Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.051 2020.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency, pp. 610-623 2021.
Giovanni Bonetta, Rossella Cancelliere, Ding Liu, and Paul Vozila. Retrieval- augmented transformer-
xl for close-domain dialog generation. arXiv preprint arXiv:2105. 09235,, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Brian L. Browning, Ying Zhou, and Sharon R. Browning. one-penny imputed genome from next-generation reference panels. American Journal of Human Genetics, 2018a. doi: 10.1016/j. ajhg.2018.07.011
Brian L Browning, Ying Zhou, and Sharon Browning. A one-penny imputed genome from next-generation reference panels. The American Journal of Human Genetics, 103(3):338-348, 2018b.
Yanshuai Cao, Marcus A Brubaker, David J Fleet, and Aaron Hertzmann. Efficient optimization for sparse gaussian process regression. Advances in Neural Information Processing Systems, 26, 2013.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane. Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations, 2020.
10

---

Published as a conference paper at ICLR 2023
Laura Clarke, Susan Fairley, Xiangqun Zheng- BranngBrrddle,. Ian Streeter, Emily Perry, Ernesto Lowy, Anne-Marie Tassé, and Paul Flicek. The international Genome sample resource (IGSR): A worldwide collection of genome variation incorporating the 1000 Genomes Project data. Nucleic Acids Research, 45(D1)) 09 2016. ISSN 0305-1048. doi: 10. 1093/nar/gkw822, URL https: //w.///oi////o .rgg /100gg11000 1093//ar////mm/// gkw829.
Phillip E. C. Compeau, Pavel A. Pevzner, and Glenn Tesler. How to apply de bruijn graphs to genome assembly. Nature Biotechnology, 2011. doi: 10. 10.1038/nbt.023.
Andreas Damianou and Neil D Lawrence. Deep gaussian processes. Artificial intelligence and statistics, pp. 207-215. PMLR, 2013.
Richard A Davis, Keh-Shin Lii, and Dimitris N Politis. Remarks some nonparametric estimates of density function. In Selected Works of Murray Rosenblatt, pp. 95-100. Springer, 2011.
Olivier Delalleau, Yoshua Bengio, and Nicolas Le Roux. Efficient non-parametric function induction in semi-super learning. In International Workshop on Artificial Intelligence and Statistics, pp. 96-103. PMLR, 2005.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp- 4171 1-766. Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 18858/6///1111111222 1423. URL https: //acl antto ogy- org/ /N/ 1423.
Trefor W. Evans and Prasanth B. Nair. Scalable gaussian processes with grid grid-structured eigenfunctions https///wwwwwwwwwwaaeeeess/e/e// www.or///bb///.../////// 02125.
(gD-grief), 2018. URL httpsttppp..... : / / arxiii
Leo Feng, Hossein Hajimirsadeghi, Yoshua Bengio, and Mohamed Osama Ahmed. Latent bot- tlenecked attentive neural processes. In The Eleventh International Conference on Learning Representations, 2023. URL https: h//p//peeveieeeeee wwwpaarreweeeeeeeeeoooolllyyyye wot.://foorummcyyllttteiiie
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018
Jerome H. Friedman. Greedy function approximation: gradient boosting machine. The Annals of Statistics, 29(5):1189 1232. 2001. doi: 10.1214/aos/011320444 URL https://doi... org/ 10. 21 ////////////135455555
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Tch. Neural processes. arXiv preprint arXiv:1807.01622. 2018.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP)) pp. 61 12-6121, Hong Kong, China, November 2019, Association for Computational Linguisties. doi: 10. 1/11 URL https: //acl antho ogy.og....o.....o....... wgg.gggg.....ggg 9- 633.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249 256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL https:////......... httpttts//occceeeeiiii http:///cccceeeeec/////////c mll. prw.srressseeeeeee / vv/ htl//lloo olgoooolloo 1orro1l0.. 10 html.
Anirudh Goyal, Aniket Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Mozer, and Yoshua Bengio. Coordination among neural modules through a shared global workspace. CoRR, abs/2103.01197, 2021. URL https: // wwwp.i..wgwaa../oooo 01197.
11

---

Published as a conference paper at ICLR 2023
Anirudh Goyal, Abram L. Friesen, Andrea Banino, Theophane Weber, Nan Rosemary Ke, Adria Puig- domenech Badia, Arthur Guez, Mehdi Mirza, Peter C. Humphreys, Ksenia Konyushkova, Laurent Sifre, Michal Valko, Simon Osindero, Timothy Lillicrap, Nicolas Heess, and Charles Blundell. Retrieval-augmented reinforcement learning, 2022. URL htt.tttttttt..........tttt.etttt http:///aaxvv.ooooeeeeo ab22222.... 08417.
Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with a continuous cache. arXiv preprint arXiv:1612.04426, 2016.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. 11X1v:1410.5401,200 2014.
arXiv preprint
Jake Grigsby, Zhe Wang, and Yanjun Qi. Long-range transformers for dynamic spatiotemporal forecasting. arXiv preprint arXiv:2109.12118, 2021.
Kelvin Guu, Kenton Lec, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint 2XXv:20002088009 2020.
Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18-22 1998.
James Hensman, Alexander G Matthews, Maurizio Filippone, and Zoubin Ghahramani. Meme for variationally sparse gaussian processes. Advances in Neural Information Processing Systems, 28,
2015.
Geoffrey E. Hinton. Connectionist learning procedures, 1989.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. /5556, 2022.
Pavel Izmailov and Dmitry Kropotov. Faster variational inducing input gaussian process classification. arXiv preprint arXiv: 1ot::///1..00.,, 2016.
Andrew Jaegle, Sebastian Borgeaud, Lean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107. 14795, 2021a.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver: General perception with iterative attention. CoRR, abs/2103.03206, 2021b. URL
fitts://arxive org/ abs 22003 03206.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. CoRR, abs/1907.10529, 2019. URL http:// /atp:i///rrivv.///// org/ abs / 1907 10529.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint urXiv:2001.0e0 008661 2020.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. International Conference on Machine Learning, pp. 5156-5166 5165. PMLR, 2020.
John Kendrew. The Encylopedia of Molecular Biology. John Wiley & Sons, 2009.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo,, Ali Eslami. Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
12

---

Published as a conference paper at ICLR 2023
Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Tom Rainforth, and Yarin Gal. Self- attention between datapoints: Going beyond individual input-output pairs in deep learning. CoRR, abs/2106.02584, 2021. URL https::////// /aappss///axivv../ wwwivv.iiiivv///iii//...//a.avvv orgg wwtp::wwwwww.ww//.......oo /ttp//ab./..........2
Alex Krizhevsky, lyy Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer. CoRR, abs/1810.008255, 2018. URL http::///// a www.wiiwwoo.raaoooooooo /abs////////// / 1810 00825.
Juho Lee, Yoonho Lee, Jungtack Kim, Eunho Yang. Sung Ju Hwang, and Yee Whye Teh. Boot- strapping neural processes. Advances in neural information processing systems, 33:6606-6615, 2020.
Na Li and Matthew Stephens. Modeling linkage disequilibrium and identifying recombination hotspots using single-nucleotide polymorphism data. Genetics, (65(4):2213-223)) 2003.
Yun Li, Cristen Willer, Serena Sanna, and Gonçalo Abecasis. Genotype imputation. Annual review of genomics and human genetics, 10:387-406, 2009.
Runyang Nicolas Lou, Arne Jacobs, Aryn P Wilder, and Nina Overgaard Therkildsen. A beginner's guide to low-coverage whole genome sequencing for population genomics. Molecular Ecology, 30 (83):5966-9993 2021.
Altti Ilari Maarala, Kalle Pärn, Javier Nuñez-Fontarnau and Keijo Heljanko. Sparkbeagle: Scalable genotype imputation from distributed whole-genome reference panels in the cloud. In Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, pp. 1-8, 2020.
Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-learning from kernel ridge- regression. In International Conference on Learning Representations, 2020.
Toan Q Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong, November 2-3 2019. Association for Computational Linguistics. URL org/2019 iws1t-1.17.
https: https///aalaattttaae
Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning sequence modeling. arXiv preprint arXiv:2207.044: 0479,, 2022.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pupers), pp. 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/w1/N18, 1202. URL https: //
org/ NN18 1202.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susamnah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 /2.::2//../44/ 1446, 2021.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.066 06222.24..2...... 2022.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pp. 63-71. Springer, 2003.
Simone Rubinacci, Olivier Delaneau, and Jonathan Marchini. Genotype imputation using the positional burrows wheeler transform. PLOS Genetics, 2020. doi: 10. 10.1331/journal 1009049.
13

---

Published as a conference paper at ICLR 2023
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with meeorr--aggeent neural networks. In International conference on machine learning, pp. 1850. PMLR, 2016.
Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. In Y. Weiss, B. Schölkopf, and J. Platt (eds.), Advances in Neural Information Processing Systems, volume 18. MIT Press, 2005. URL https: http///poeedngggeeeeeeeeee neurips....... wwtpscccpppppee wccppaper/205 file/44 /ete/449777800 7b1 ae www.eaabbb2222cc
Michalis Titsias. Variational learning of indueing variables in sparse gaussian processes. In David van Dyk and Max Welling (eds.). Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pp. 567-574, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16-18 Apr 2009. PMLR. URL https: httpss//pccceedingss.. mlr. press / wwtttitttteeeee.e titsiasss9 html.
Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung. Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision. CoRR, abs/2105.01601 2021.
URL https:tt///// arxiv. org/ abs/ abs//100 2205. 01601.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0477 2020.
Yali Wang, Marcus Brubaker, Brahim Chaib- Draa, and Raquel Urtasun. Sequential inference for deep gaussian process. Artificial Intelligence and Statistics, pp. 694-703, PMIR, 2016.
Andrew Wilson and Hannes Nickisch. Kernel interpolation for scalable structured gaussian processes (kiss-gp). In International conference on machine learning, pp. 1775-1784. PMLR, 2015.
Andrew Gordon Wilson, Christoph Dann, and Hannes Nickisch. Thoughts massively scalable gaussian processes. arXiv preprint 1 2015.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
Sam Wiseman and Karl Stratos. Label-agnostic sequence labeling by copying nearest neighbors. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 5363-5369, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19- 1533. URL https: / / /aacalanii ccllnnhhlooggg httpl/lgy..oooo11o//ogog
Wrayner. Human omni marker panel. URL https: strand/.
ox. ac. uk/ uk/-wrayner/
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Víkas Singh. Nyströmformer: A nystöm- algorithm for approximating self self-attention. In Proceedings of the... AAAI Conference on Artificial Intelligence. AAAI Conference Artificial Intelligence, volume 35, pp. 14138. NIH Public Access, 2021.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse. explainable multi-hop question answer- ing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369 2380, Brussels, Belgium, October-November 2018. Association for Computational Lin- guistics. doi: 10.18653/v1/DD 1259. URL https: /tattss:/acclae http:///aaaaattttaaaa/ee hwwp:/hhwwoooooooogoooooo o
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslic, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, al. Big bird: Transformers for longer sequences. Advances in Newral Information Processing Systems, 33:17283- 17297, 2020.
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey Hinton. Lookahead optimizer: k steps forward, step back. Advances in Neural Information Processing Systems, 32, 2019.
14

---

Published as a conference paper at ICLR 2023
APPENDIX: SEMI-PARAMETRIC INDUCING POINT NETWORKS AND NEURAL PROCESSES
A EXPERIMENTAL DETAILS
A.1 COMPUTE RESOURCES
We use 24GB NVIDIA GeForce RTX 3090, Tesla V100-SXM2-16GB and NVIDIA RTX A6000- 48GB GPUs for experiments in this paper. A result is reported OOM if it did not fit in the 24GB GPU memory. We do not use multi-GPU training or other memory-saving techniques such as gradient checkpointing, pruning, mixed precision training, etc. but note that these are orthogonal to our approach and can be used to further reduce the computational complexity.
A.2 TRAINING OBJECTIVE
We define a binary mask matrix for given sample as M |m ()) mm ... ....mm mmm1) where ( for labels and I = p for attributes. Then the loss over labels and attributes for each sample i is given by.
(labels.(0) (y (i) (1) pred Y Viues M la)e
Σ
(i) mi Llypood..
(.ode..Imme..) yime.s)
prod.
j=1
Cantmbutes,(1) (x (xpedd Xinee M Mattmbuness(1) (+))
(i)
Σ
m (6)
[(x xun..))
L(x
where C(y pred.j )) 2 y
iue.icc log(softmax ( pred...c)) Cross Entropy Loss for C-way
(1)
Classification and Cly pred.j y mme.)) (y
me, y p )2 for MSE Loss. L(x x ximeej) for
attributes that are reconstructed is computed analogously
For chunk masking, fraction p of the samples selected have the mask matrix for labels ( 1
M = 1,
with probability p
0, otherwise
A.3 GENOMIC SEQUENCE IMPUTATION
Imputation is performed on single-nucleotie polymorphisms (SNPs) with corresponding marker panel specifying the microarray. We randomly sample five sections of the genome for chromosome 20 for conducting experiments. Each section is selected with 100 SNPs to be predicted and 150 closest SNPs are obtained. For compact encoding of SNPs, we form K K-mers, which are commonly used in various genomies applications (Compeau et al., 2011) 1), where K is syper-parameter that controls the granularity of tokenization (how many nucleotides are treated as a single token). This now becomes 2= -way classification task. We K to 5 for all the genomics experiments, (100/5) target SNPs to be imputed and 30 (150/5) attributes per sampled section. We report pearson R² for each of the five sections in Table with error bars per window for five different seeds. For computational load, we report peak GPU memory usage in GB where applicable, an average of train time per epoch in seconds, and parameter count per method. Table 8 provides Pearson R²2 for each of the 10 regions using a single model, thus learning the Genotype imputation algorithm.
that there are 20
In Table 9, we analyze the effect of increasing reference haplotypes during training on pearson R² computed by NPT and SPIN. The reference haplotypes in the train dataset are gradually increased from a small fraction of 1% to 100% available. Pearson R² reported cumulatively for 10 randomly selected regions with window size=300. We observe that the performance for both SPIN and NPT improves with increasing reference dataset. However, NPT cannot be used beyond a certain set of reference samples due to its GPU memory footprint, while SPIN yields improved performance.
15

---

Published as a conference paper at ICLR 2023
Hyperparameters In Table 10, provide the range of hyper-ppanee parem-erreeterss that grid searched for different methods. Beagle is a specialized software using dynamic programming and does not require any Hyper-parameter from the user.
Table 7: Performance on Genomics Imputation
Pearson
Peak GPU
Params
Avg. Train time/epoch(s) |
Approach
R²
Mem (GB) |
Count
Genomics Dataset (SNPs 68300-68400)
GBT Traditional ML MLP
81.12
97.63
KNN
86.96
Bio Software
Beagle
98.07
NPT-16
96.96 +0.28
0.45
16. 76
2.22
Transformer
STF-16
97.02 +
0.76
33.4M
2.93
SPIN-16
97.13 +0.21
0.28
8.1M
2.23
Genomics Dataset (SNPs 169500-169600)
GBT Traditional ML MLP
91.53
97.19
KNN
95.65
Bio Software
Beagle
97.87
NPT-16
97. +:008
0.45
16. 16.7M
1.98
Transformer
STF-16
98.07 +0.15
0.76
33.4M
2.99
SPIN-16
97.50 #0.12
0.28
8.1M
2.76
Genomics Dataset (SNPs 287600-287700)
GBT
81.12
Traditional ML MLP
96.20
KNN
95.56
Bio Software
Beagle
92.62
NPT-16
97.07 +:0066
0.45
16. 76.7M
2.24
Transformer
STF-16
97.09 +0.07
0.76
33.4M
2.99
SPIN-16
97.11 #0.04
0.28
8.1M
2.60
Genomics Dataset (SNPs 424600-424770)
GBT Traditional ML MLP
82.77
91.98
KNN
84.39
Bio Software
Beagle
93.72
NPT-16
93.49 +0.70
0.45
16.7M 7M
2.23
Transformer
STF-16
93.38 10.90
0.76
33.4M
2.90
SPIN-16
93.83 #0.65
0.28
8.1M
2.21
Genomics Dataset (SNPs 543000-543100)
GBT
72,66
Traditional ML MLP
89,56
KNN
78.22
Bio Software
Beagle
94.58
NPT-16
91.30 +1.14
0.45
16.7M
2.35
Transformer
STF-16
91.49 10.39
0.76
33.4M
2.52
SPIN-16
91.36 +0.18
0.28
8.1M
2.28
16

---

Published as a conference paper at ICLR 2023
Table 8: Pearson R 1 for each region for the Multiple Windows Experiment
Small
Medium
Large
Runs
SPIN
NPT
SPIN
NPT
SPIN
NPT
1
97.34
96.85
94.30
93.86
97.55
OOM
2
98.09
97.80
83.82
81.70
90.38
3
96.98
97.13
97.04
96.91
87.46
4
93.39
93.61
95.46
95.24
95.93
5
92.12
91.66
93.19
92.53
97.05
6
95.59
94.39
94.39
93,99
91.30
7
94.58
93.86
89.97
88.63
93.66
8
93.34
92.67
93.70
93.39
94.48
9
85.01
85.82
94.30
93.54
94.39
10
97.70
97.31
95.52
94.80
87.94
Total
95.39
95.06
93.55
92.89
93.90
OOM
Table 9: Cumulative Pearson R² t for 10 randomly selected genomic windows
Reference Samples
Pearson R
Peak GPU (GB)
SPIN
NPT
SPIN
NPT
44 (1%)
84.87
85,00
8.64
18.07
219 (5%)
86.25
85.54
8.77
18.43
658 (15%)
87.55
86.35
9.1
19.69
1316 (30%)
90.33
9.59
OOM
4388 (100%)
92.91
12.83
OOM
Table 10: Hyperparameters for Genomics Dataset
Model
Hyperparameter
Setting
Embedding Dimension Depth
[16, 128]
[2, 8]
Label Masking
[0, 0.5]
NPT, SPIN,
Target Masking
[0.3]
Set Transformer
Learning rate
[le 5, le - 2]
Dropout
[0.4, 0.6]
Batch Size
[256, 5008 (No Batching)
Inducing points
[3,100]
Max Depth
[5, 10]
Gradient Boosting
n Lestimatorrs Learning rate
[100]
[le 2]
Hidden Layer Sizes
[(500, 500, 500)]
Batch Size
[128, 256]
MLP
L2 regularization
[0, 1c - 2]
Learning rate init
[1c le
n neighbors
[2, 1000] [distance] [auto]
weights
KNN
algorithm
Leaf Size
[10, 100]
Bio Software
None
None
17

---

Published as a conference paper at ICLR 2023
A.4 UCI REGRESSION TASKS
In Table 11, we report results for 10 cross-validation (CV) splits for Yacht and Concrete datasets, CV splits for Boston-Housing datasets, and CV split for Protein dataset. Number of splits were chosen according to computational requirements. Below we provide details about each dataset.
Yacht dataset consists of 308 instances, continuous, and categorical features.
Boston Housing dataset consists of 506 instances, 11 continuous, and 2 categorical features.
Concrete consists of 1030 instances, and continuous features.
Protein consists of 45,730 instances, and 9 continuous features.
Table 11: Performance on UCI Regression Datasets
Peak GPU
Params Count
Avg. Train time/epoch(s) +
Approach
RMSE
Mem (GB)
Boston-Housing
GBT
3.44+:022
Traditional MLM
3.32-10.39
KNN
4.27:10.37
NPT 2.92:10.15
8.2
168.0M
1.45
Transformer
STF
3.33+11.33
16.5
336.0M
1.99
SPIN
3.01 +0055
6.3
127.2M
1.63
Yacht
GBT 0.87-10.37
Traditional MLMLP
0.83110.18
KNN
11.97+2.06
NPT
1,42:00.4
2.1
42.7M
0.10
Transformer
STF
1.29440.34
4.1
85.4M
0.19
SPIN
1.28+-0.66
1.6
32.2M
0.07
Concrete
GBT 4.61:10.72
Traditional MLMLP
5.29+00774
KNN
8.62++0.77
NPT
5.21+0.20
3.4
69.9M
0.13
Transformer
STF
5.35++0.80
6.8
139.9M
0.21
SPIN
5.17+0.87
1.9
39.4M
0.21
Protein
GBT
3.61
Traditional MLMLP
3.62
KNN
3.77
NPT
3.34
13.1
86.
18.13
Transformer
STF
3.39
5.3
172.3M
8.34
SPIN
3.31
3.2
43.0M
24.28
A.5 UCI CLASSIFICATION TASKS
In Table 12, we report results for 10 CV splits for Breast Cancer dataset and 1 CV split for Kick, Income, Forest Cover, Poker-Hand and Higgs Boson datasets. Number of splits were chosen according to computational requirements. Below we provide details about each dataset.
NPP reports mean of 1.27 on this task that could not reproduce. However, we emphasize that for UCI experiments, all the model parameters kept for all the transformer methods.
18

---

Published as a conference paper at ICLR 2023
Breast Cancer dataset consists of 569 instances, 31 continuous features, and 2 target classes.
Kick dataset consists of 72,983 instances, 14 continuous and 18 categorical features, and target classes
Income consists of 299,285 instances, 6 continuous and 36 categorical features, and target classes.
Forest Cover consists of 581,012 instances, 10 continuous and 44 categorical features, and
7 target classes.
Poker-Hand consists of 1,025,010 instances, 10 categorical features, and 10 target classes.
Higgs Boson consists of 11,00,,000 instances, 28 continuous features, and target classes.
We provide the range of hyperparameters for UCI datasets in Table 13. Additionally, we provide aver- age ranking separated by Regression and Classification tasks in Table 14 and Table 15, respectively.
19

---

Published as a conference paper at ICLR 2023
Table 12: Performance on UCI Classification Datasets
Peak GPU
Params Count +
Avg. Train
Approach
Accuracy
Mem (GB) J
time/epoch(s)
Breast Cancer
GBT
94.03=:2.74
Traditional ML MLP
94.03=+3.05
KNN
95.26.12.48
NPT 95.7911.22
2.6
51.3M
0.15
Transformer
STF
94.91
5.2
102.6M
0.21
SPIN
96.32+1.54
0.9
16.9M
0.18
Kick
GBT
90.20
Traditional ML MLP
89.96
KNN
87.71
NPT
90.04
14.9
232.6M
56.22
Transformer
STF
90.03
15.0
465.0M
52.35
SPIN
90.06
3.6
73.7M
27.76
Income
GBT
95.8
Traditional ML ML.
95.4
KNN
94.8
NPT
95.6
24
1504M
Transformer
STF
OOM 11.5
SPIN
95.6
418.9M
68.02
Forest Cover
GBT 96.70
Traditional MLMPP
95.20
KNN
90.70
NPT 96.73
18.0
644.7M
230.47
Transformer
STF
OOM 5.4
SPIN
96.11
162.7M
138.38
Poker-Hann Hand
GBT
78.71
Traditional ML MLP
56.40
KNN
54.75
NPT
80.11
9.8
104.0M
93.56
Transformer
STF
79.89
3.1
52.1M
83.13
SPIN
82.98
1.7
11.8M
72.05
Higgs Boson
GBT 76.50
Traditional MLMLP
78.30
KNN
NPT 80.70
14.7
179.5M
1,569.39
Transformer
STF
80.48
12.8
359.0M
1,796.94
SPIN
80.01
4.9
62.
983.44
20

---

Published as a conference paper at ICLR 2023
Table 13: Hyperparameters for UCI Dataset
Model
Hyperparameter
Setting
Embeddiin Dimension
[16, 128]
Depth
[8]
Label Masking
[0, 0.5]
NPT, SPIN,
Target Masking
[0.3]
Set Transformer
Learning rate
[1e 5, le 2]
Dropout
[0.4, 0.6]
Batch Size
[2048, No Batching]
Inducing points
[5,10]
Max Depth
[3, 10]
Gradient Boosting
n_estimators
[50, 1000]
Learning rate
[le - 0.3]
Hidden Layer Sizes
[(25) - (500), (25, 25) (500, 500),
(25, 25, 25) (500, 500, 500)
MLP (Boston Housing Breast Cancer, Concrete, and Yacht)
Batch Size
[32, 256] [0. 1]
L2 regularization Learning rate Learning rate init
[constant, invscaling, adaptive]
[1e - 5, 1e 1
Hidden Layer Sizes
[25,, 25, 25) (500, 500, 500)]
Batch Size
[128, 256]
MLP (Kick, Income)
L2 regularization
[0, le 2]
Learning rate Learning rate init
[constant, invscaling, adaptive]
[le 5,
n_neighbors
[2, 100J
KNN (Boston Housing
weights
[uniform, distance]
Breast Cancer, Concrete, and Yacht)
algorithm
[ball tree, kd tree, brute]
Leaf Size
[10, 100]
neighbors
[2, 1000] [distance] [auto]
weights
KNN (Kick, Income)
algorithm
Leaf Size
[10, 100]
Table 14: Average Ranking on UCI Regression Dataset (Yacht, Boston Housing, Concrete, Protein)
based on RMSE
Peak GPU Mem
Approach
Average Ranking order
(relative to NPT).
GBT
3.00
Traditional ML
MLP
3.25+1.71
KNN
6.00.10.00
NPT
2.75
1.0x 1.71-0.48x 0.65-0.09x
Transformer
STF
4.00.10.82
SPIN
2.00+-0.82
21

---

Published as a conference paper at ICLR 2023
Table 15: Average Ranking on UCI Classification Dataset (Breast Cancer, Kick, Income, Forest Cover, Poker-Hand, Higgs-Boson based on Classification Accuracy
Peak GPU Mem
Approach
Average Ranking order
(relative NPT)
GBT
3,00:1.90
Traditional ML
MLP
4.67:10.82
KNN
5.00.11.22
NPT
2.0010.89
1.0x 1.05=0.70x
Transformer
STF
3.25++0.96
SPIN
2.17+0098
0.31=0.100
A.6 NEURAL PROCESSES
Variational Lower Bound follows:
The variational lower bound objective used to train latent NPs is as
D
CIPNP,ELB
log pa (y: (4i) z, D(d), x xxx (di)) + KL(g(z) D. D( p(z D[
D
d=1
1=1
where KL is the Kullback-Leibler divergence, q(2 D( (( ) is the posterior distribution condi- tioned on target and context sets, and y(z (! the prior conditioned only on the context.
Hyperparameters Learning rates for all experiments were 54 with Cosine Annealing learning rate scheduler applied. Model parameters were optimized using the ADAM optimizer (Kingma & Ba, 2014).
Architectures In Table 16 and 17, we detail the architecture for the conditional NPs (CANP, BANP, CIPNP) and latent variable NPs (ANP, IPNP) used in Section 4.2, respectively. Note that although the conditional NPs do not have latent path, in order to make them comparable in terms of number of parameters we add another deterministic encoding Pooling Encoder to these models, as described in Lee et al. (2020). In these tables, we remark where XABD is used as opposed to regular self attention between context data points. use the following shorthand notation below: X, are context features for batch of datasets stacked into tensor of size B x x 1, and X, is defined similarly. D denotes the full dataset, features and labels for both context and target, stacked into a tensor of size 1.
B x
Note, although the equations described in Section 3.3 and in Table and 17 use tensors of order 4, in practice attention is performed along the correct dimension (i.e., data points).
tensors of order and permute the dimensions of the tensor in order ensure that
Finally, in Figure 5, we provide more detailed diagram of the (conditional) IPNP architecture, which excludes the additional Pooling Encoder.
22

---

Published as a conference paper at ICLR 2023
Y
Fo Induced Latent
Encoding
y:
Figure 5: CIPNP architecture diagram
Table 16: CANP BANP CIPNP architecture (no latent path)
Input
Layer(s)
Output
Cross Attention Encoder
X, E Bxnxixl
MLP (1 hidden layer, ReLU activation)
Q E R Bxxx128x1
X, R Bxmxixl xxl
MLP hidden layer, ReLU activation)
K Bxmx 128
D E Bxxx2x
(1) MLP (1 hidden layer, ReLU activation),
(2) MAB(D, D) (Self attn between data points) for CANP/BANP
MAB(HD. D) (XABD) for CIPNP
V Ic E R BBxmx128x1 r R. Bx
Q.K.V
Cross attn between query and context points
Pooling Encoder DE R Bxmx2x1
(1) MLP (1 hidden layer, ReLU activation),
(2) MAB(D, D) (Self attn between data points)
for CANP/BANP
MAB(H D) (XABD) for CIPNP
(3) Mean pooling (on context points)
(4) MLP (I hidden layer, ReLU activation)
(5) Repeat n times
r' E R Bxxx12881
Decoder
r,, E Bxax2577
(1) FC
(2) MLP (2 hidden layer, ReLU activation)
Ø E R Bxax2
Ø M,
chunk (splits input into tensors of equal size) Sampler
14, E Bxnx1
Y, R Bxax1
23

---

Published as a conference paper at ICLR 2023
pts pred 17 sti
Cbx:
ctx pts
prod moon
+/ std
cox pts pred mean +) std
Cbx:
Figure 6: Predicted mean standard deviation of y for different NP models given varying context sizes: 4 (top), 8 (middle), and 16 (bottom)..
Table 17: ANP IPNP architecture (latent path)
Input
Layer(s)
Output
Cross Attention Encoder
Same CANP/ BANP/ CIPNP (Table 16)
Latent Pooling Encoder
D E TR. Bxmx2x1
(1) MLP (1 hidden layer, ReLU activation),
(2) MAB(D, D) (Self attn between data points)
for ANP
MAB(Hp, D) (XABD) for IPNP
(3) Mean pooling (on context points)
(4) MLP (1 hidden layer, ReLU activation)
0 E IR Bx266
Φx 11, Ez
chunk (splits input into tensors equal size)
P1z En E R Bx128x1
(1) Sampler
(2) Repeat n times
z E R Bxxx112xx N( 2i))
Decoder
rI, E Bxxxx225xx1
(1) FC
(2) MLP (2 hidden layer, ReLU activation)
& E Bxxxx
chunk (splits input into 2 tensors of equal size) Sampler
μ. E E Bxnx1
μ, Σ
Y, E R Bxxx1
N E2)
Qualitative Uncertainty Estimation Results In Figure 6, we show baseline and inducing point NP models trained with context sizes [64, 128] and display the output of these models on new datasets with varying numbers context points (4, 8, 16). We observe that the CIPNP and IPNP models better capture uncertainty in regions where context points have not been observed.
To provide more quantitative results of how well our NP models capture uncertainty relative to baselines. take models trained with context sizes [64, 128] and evaluate them on 1,000 evaluation batches each with number of targets points ranging from 4 to 64. We repeat this experiment three times with varying numbers of context points (4, 16) available for
Quantitative Calibration Results
24

---

Published as a conference paper at ICLR 2023
Cbx:
Ctx: 8
Ctx: 16
1.0
Perfect
0.8
CANP
BANP
of values in
ANP
0.6
CIPNP
0.4-
0.2
0.0
0.0
0.2
0.8
0.0
0.4 0.5
1.0
0.2
0.4
0.8
1.0
Confidence Interval (CI)
Figure 7: Calibration of NP models given varying context sizes: 4 (left), 8 (middle), and 16 (right).
Table 18: Calibration scores (4) for NP models across context sizes. Calibration score equals mean squared deviation from 45 line of number of target points falling within confidence intervals ranging from 0 to by intervals of 0.1, see Equation (1). Best (lowest) scores for each context size are bolded.
Calibration score
Context
CANP
BANP
ANP
CIPNP
IPNP
4
0.065
0.062
0.065
0.006
0.023
8
0.025
0.024
0.026
0.002
0.004
16
0.006
0.005
0.005
0.011
0.008
each evaluation batch. In Figure 7, we see that in lower context regimes, CIPNP and IPNP models are better calibrated than the other baselines. As context size increases, the calibration of all the models deteriorates. This is further reflected in Table 18, where we display model calibration scores. Letting CT be confidence intervals ranging from 0 to 1.0 by intervals of 0.1, PC7 be the fraction of target labels that fall within confidence interval CT, and n be the number of confidence intervals, this calibration score equal to:
Σ - TI CT=0
(I)
This score measures deviation of each model's calibration plot from the 45" line, Future work will explore the mechanisms that enable inducing point models to better capture uncertainty.
A.7 ABLATION ANALYSIS WITH SYNTHETIC EXPERIMENT
We formulate a synthetic experiment where the model can only learn via XABD layers. First, we initialize a random binary matrix with 50% probability I's, number of rows=5000, and number of columns=50. We set the last 20 columns to be target labels. Next, we copy section of the dataset and divide it into three equal and disjoint parts for train query, val query, and test query. Since there no correlation between the features and the target, the only way for the model to learn is via XABD layers (for small dataset the model can also memorize the entire training dataset). This similar to the synthetic experiment in NPT (Kossen et al., 2021). except that there is no relation between the features and target in our setup. We find that both default SPIN and SPIN with XABD only component achieves 100% binary classification accuracy, whereas SPIN with XABA only component achieves 70.01 % classification accuracy, indicating the effectiveness XABD component.
A.8 QUALITATIVE ANALYSIS FOR CROSS-ATTENTION
In order to understand what type of inducing points are learnt by the latent Hp, we formulate a toy synthetic dataset as shown in Figure 8. We start with two centroids consisting of binary strings with 120 bits and add bermoulli noise with p 0.1. We create the labels as another set of 4 bit binary strings and apply bernoulli noise with p 0.1. In this way create a dataset with datapoints belonging to two separate clusters. Figure (a) shows the projection this dataset with two principal
25

---

Published as a conference paper at ICLR 2023
map for Xaury
Diglcam
Aux Train
Query
Train Query
Quary
Masked Labels
Decoder cross attontion map, with samples sorted by cluster labels
Dataset Generatin
Figure 8: Synthetic Experiment analyzing cross- ansssttttntioo:: (a)-(b) Data generating process to form two distinct clusters and data matrix with duplicate query samples and their labels masked. (c) Cross-attention map between query samples and the latent HD
components when projected with PCA, highlighting that the dataset consists of two distinct clusters. We duplicate a part of this data to form the query samples so that they can be looked up from the latent via the cross attention mechanism. Figure 8 (b) shows the schematic of dataset with masked values for query labels. We use a SPIN model with a single XABD module, two induced latents (/=2) and input embedding dimension of 32 and inspect the cross-attention mechanism of the decoder. In Figure (c), we plot the decoder cross attention map between test query and the induced latent and observe that the grouped query datapoints attend the latents consistent with the data generating clusters.
A.9 IMAGE CLASSIFICATION EXPERIMENTS
We conducted additional experiments comparing SPIN and NPT on two image classification datasets. Following NPT, compare the results for image classification task using a lincar patch encoder for MNIST and CIFAR10 dataset (which is the reason for the lower accuracies compared to using CNN-based encoders). Table 19 shows that for the linear patch encoder, SPIN and NPT both perform similarly in terms of accuracy, but SPIN uses far fewer parameters.
Table 19: Image Classification Experiments
Dataset
Approach Classification Accuracy Peak GPU (GB)
Params Count
NPT
97.92
1.33
33.34M
MNIST
SPIN
97.70
0.37
8.68M
NPT
68.20
18.81
900.36M
CIFAR-10
SPIN
68.81
5.13
217.53M
A.10 EFFECT OF NUMBER OF INDUCED POINTS
We conducted sensitivity analysis of SPIN's performance with respect to h and f and found that SPIN is fairly robust to the choice of these syper-parameters, evidenced by the low standard deviations in Table 20. This reflects redundancy in data and why attending to the entire dataset is inefficient.
Table 20: Effect of induced points h f for one genomic window (SNPs 424600-424700)
Induced Points h
Induced Points f
Pearson R
Peak GPU (GB)
(5 (5.
10
94.03 10.50
0.28 +0.
10
30)
94.15 +0.25
0.32 +0.05
30)
30)
94.03 =0.28
0.32 #0.05
B COMPLEXITY ANALYSIS
We provide time complexity for one gradient step, with the number of layers, batch size b equal to training dataset size n during training, and one query sample during inference for transformer methods in Table 21. There are two operations that contribute heavily the time complexity. First
26

---

Published as a conference paper at ICLR 2023
computation of Q.K", second is the four times expansion in the feedforward layers. For NPT, the time complexity is given by maximum of ABD, ABA, and four times expansion in feedforward layers during ABD, that is max(n,n' de, ninde, e, An e²) during training and inference. Set Transformer consists of ISAB blocks that perform one cross-attention between latent and dataset to project into smaller space and cross attention between dataset and latent to project back into input space for each layer. This results in complexity that is max(2nyndf e, 2nmhh fe, 8nind during training and inference. For SPIN, the time complexity is given by maximum of XABD, XABA, ABLA, four times expansion in feedforward layers and one cross- aotsentntniionn for Predictor module. This can be formulated max( max(nimhff fe, nindfe, nin.ee, An, fee, nhde, 4nd2 At inference, SPIN only uses the Predictor module, with the resultant complexity as max(hde, 4d"e²).
We note that during training for NPT, if >4de, then Q.KT computation in ABD dominates, otherwise the four times expansion of feedforward for ABD dominates. For Set Transformer, usually the four times expansion of feedforward dominates. For SPIN, depending on values for ny, d, f, h, e, different computations can dominate, however it is always linear in dataset size During inference SPIN's time complexity is independent of number layers ny and dataset size n and depends entirely on inducing datapoints h, model embedding dimension e, and feature+ttaget space d.
Table 21: Time Complexity
Time Complexity
Approach
Train
Test
max de, de, ninde, Anqud e²)
max de, nndee e, Anqud e
NPT
max (2mmnffe e, 2nmhffe e)
max (2nimdf e. 2nynhfe, 8nind?
STF
max(runhfee fe, nendf e, mmnfee SPIN
max(hde, 62
Anguf2, nhdde And e))
C
CODE AND DATA AVAILABILITY
UCI and Genomic Task Code duced from here.
The experimental results for UCI and genomic task can be repro-
Neural Processes Code The experimental results for the Neural Processes task can be reproduced from here.
Data for Genomics Experiment The vef file containing genotypes can be downloaded from 1000Genomes chromosome 20 vef file. Additionally, the microarray used for genomics experiment can be downloaded from HumanOmni2.5 microarray. Beagle software, used as baseline, can be obtained from Beagle 5.1.
UCI Datasets
All UCI datasets can be obtained from UCI Data Repository.
'The complexity at test time for SPIN is max(n/de, 4nd when either using the optional cross- gross-attention in the predictor module or when the encoder is enabled at test time, such as in the multiple windows genomic experiment.
27


Reviews:
[{"review_id": "v2JZNIOgE2", "reviewer": "Reviewer_jnqW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper develops a special kind of neural network architecture that the authors call SPIN, semiparametric inducing point networks.  This architecture takes a dataset as input and using a \n“dataset encoder” outputs a latent set of “inducing points” which one could imagine either as pseudo-examples representing the larger dataset or summary statistics describing the data.  These inducing points are incorporated into a prediction network via cross-attention layers that effectively compute dot products between a query and the inducing points.  The authors demonstrate the efficacy of this approach empirically on the UCI data, a meta learning setup and a genotype imputation problem.", "summary_of_the_review": "Overall, I think this paper represents an interesting innovation that seems novel and is well described.  I think the paper missed the mark a bit in terms of formulating / evaluating the method as a stochastic process.  The experiments are useful as a proof of concept and the genotype imputation experiment presents a scenario where this model seems to make sense over standard deep learning approaches.  The UCI experiments are a bit weak in terms of the baselines and the overall results.  Overall I think the novelty and presentation warrant an accept but the paper could be stronger.  I would vote accept but I wouldn’t champion the paper strongly (e.g. for a presentation).  \n\n", "strength_and_weaknesses": "Strengths:\nThe paper is clear and well written.\nThe proposed model seems intuitive, scalable and effective.\nI like that the authors found multiple problem domains where their proposed approach would outshine traditional GP or deep learning approaches.  Genomic imputation is neat and seems like an important real world application.\nThe method seems to work well in the experiments presented.\nThe GPU memory required by the method seems clearly less than the stronger baseline NPT.  \n\nWeaknesses:\nA major aspect of the neural process literature is that the models are a stochastic process, i.e. they induce a distribution over functions conditioned on some observed data.  This paper completely ignores that part and considers only predictive accuracy.  For example, a major claimed advantage of neural processes is that they can provide a high quality estimate of uncertainty away from the data.  This paper completely ignores the “processes” part of neural processes and evaluates the models as if they’re standard deep networks on standard deep learning benchmarks (agreed genomic imputation is non-standard and more exciting). \nThe treatment of existing work is a little sloppy.  I’m not sure why inducing points and deep kernel learning are attributed to Wilson & X.  There’s a deep literature on both subjects and these papers are one contribution of many in each subfield (neither the first, last or SOTA).\nThe choice of baselines seems quite weak.  Particularly, UCI has been completely and utterly annihilated by Bayesian deep learning papers.  Take a look at all the papers citing Bayesian dropout (Gal & Gharhamani).  The fact that the baselines here are: a method from 2001 - GBT, K-Nearest Neighbors and “Multilayer Perceptron” seems quite suspect given that one can just extract the numbers from recent papers.  Why are all the baselines from 20 years ago or more?  That is compounded by the fact that only a ranking is provided, so a reader can’t directly compare to more recent literature.  I found the results in the appendix and confirmed that 1) in general they’re not as strong as dropout (which is taken to be a straw man in many papers) and 2) the results often don’t seem statistically significant.  Could the authors provide some more context about why they chose these baselines and the significance of the ranking results?\nThe ranking results are just not statistically significantly better than NPT.  I think it’s ok to say they are competitive with NPT but require less than half the memory, but it doesn’t seem justified to say that they are better on average.\nIf the claim is that SPIN / IPNP is faster, then plot with walltime as an axis.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mGTLHZ5-8ey", "reviewer": "Reviewer_wVgc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new semi-parametric learning method called SPIN. It builds upon non-parametric transformers (NPT), which use attention between training points to learn compact and effective predictors at the cost of quadratic runtime in the number of training points considered (context size). SPIN addresses this by utilizing inducing points, a small constant number of vectors summarizing the large number of training points. The resulting architecture scales linearly in the context size at training time, and linearly only in the selected number of inducing points at test time. It is demonstrated that depending on the selected context size, SPIN yields improved prediction performance, reduced computational cost, or both. Additionally, SPIN is employed for metalearning on synthetic and genome imputation data, where it delivers improved/competitive performance, respectively.", "summary_of_the_review": "In summary, I think the paper represents a clear step forward for semi-parametric methods. The idea of using inducing points is well executed and thoroughly evaluated. The meta-learning and genome imputation experiments push the applicability of such methods to new domains. I therefore recommend acceptance.\n\n-----\n\nPost Rebuttal Update: After reading the other reviews and the authors' responses, I maintain my positive view of the paper.", "strength_and_weaknesses": "Strengths:\n - The paper is well motivated, as it addresses a clear weakness of NPT in a suitable and convincing manner.\n - The method is clearly explained and sensibly constructed.\n - The experimental evaluation is thorough and highlights important aspects of the method, including its sensitivity to context size, its parameter count, and its memory usage.\n - The improvements over NPT on UCI and meta learning experiments are quite convincing.\n\nWeaknesses:\n - On the genome imputation experiment, there is no significant performance increase over prior methods, the result is only notable in terms of SPIN's generality and low parameter count. Maybe some more concretely desirable property such as inference time could be measured here for a more convincing argument.\n - The resolution of the figures, especially figure 3, should be improved, ideally by making them a vector graphics.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "f9LWbH9S-C-", "reviewer": "Reviewer_TnrL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a mechanism for semi-parametric prediction using neural attention mechanisms and a variant of inducing points.\n\nIn this work, the authors combine ideas such as cross-attention between attributes and between datapoints, yielding a framework -denoted SPIN- which given a new dataset produces a set of lower dimensional inducing points H in an encoder structure.\nThis set H, can either be used to update to the next set H given data, or it can be used in a prediction layer in a comnstruct which allows performing predictions of the style p(y|x, H).\n\nThe authors propose to then utilize SPIN for a neural process variant, which using these inducing points is able to incorporate large context sizes in context datasets D_c to make predictions on query datasets.\n\nIn their experiments, they show competitive performance on various tasks related to the neural process literature where they exhibit strong scalability due to better memory utilization of their method compared to useful baselines.\nOf particular note is an example for genotype imputation, which seems to benefit strongly in this meta-learning scenario from the proposed methodology.", "summary_of_the_review": "The authors present an evolution and marriage of the streams of work on neural processes and cross-datapoint attention, which they bring together to propose an architecture which can utilize large context sets to perform semi-parametric prediction.\nI really enjoyed the key application to genotype imputation as an example that would be uniquely enabled by this model.\n\nOverall the paper is of good enough quality and interest and although it does not meaningfully extend or propose much technical novelty or deep insights into why the utilized pieces here work as well as they do, the empirical qualities regarding scalability and positioning of the paper appear and useful and valuable.\n\nIf the authors were able to better explain and understand the interplay of their layers and which computations they are amortizing I would have been more excited.", "strength_and_weaknesses": "Strengths:\nThe technique the authors propose appears to work well empirically.\nWhile the core setup is a familiar one of neural processes, the particular handling of inducing points here allows the models to remain competitive and scale beyond the typical comparators, which is a promising avenue for its future use as one may want to incorporate larger context datasets D_c.\n\nThe authors do a good job comparing to a few benchmarks such as UCI datasets, GP-prior modeling, and genotype imputation.\n\nWeaknesses:\nSome brief ablation experiments seem to indicate that the lion;s share of the work is carried by the cross-data attention modeling part here. It would be great to get deeper insight into this.\nIn general the work appears to propose an architecture which feels somewhat \"engineered\" and while the authors show that it works, it would be great to understand better why each part works, how sequencing these different layers makes sense , what calculation the model is really amortizing over here, and deepen the analysis of the ablations.\nWhat does the ABLA layer really do? I understand the authors claim it refines H_A, but it is not clear to me what this is amortizing and the work does not really convey the intuition here.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3TYU1IZyRCh", "reviewer": "Reviewer_GJez", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces semi-parametric inducing point networks, which learn a latent representation of the entire training set which can be queried at inference time efficiently. The authors experiment on the UCI benchmarks, a synthetic meta-learning task, and a genomic imputation task. The key contribution is a decrease in computational cost vs existing methods.", "summary_of_the_review": "Overall, I think this is a solid paper and it encourages future work on compressing training sets into queryable encodings. The key contribution seems to be the computational efficiency of their method. It's clear that it's more computational efficient at the same performance, but it's not very clear how important computational efficiency is here.", "strength_and_weaknesses": "Strengths:\n\nThis paper is excellently written. I enjoyed reading it - particularly the intro and background, and it flows nicely.\n\nOverall, the results are slightly better than existing methods. But SPIN requires significantly less GPU usage and param count than existing methods, as well as allows for larger context size. Computational cost grows linearly in the training set size compared to  the quadratic growth of existing methods.\n\nThe experiments seem convincing, and I particularly like the genomic imputation experiments. This type of model seems like a good method for genomics, where we always have the reference genome and prior lab-experiments.\n\n\nWeaknesses:\n\nThe motivation for why we need more GPU efficient methods compared to NPT isn't very clear. Table 2 shows the effect of a larger context size that NPT can't fit in memory, but aside from that I didn't see anything else. For something like genomic imputations, those aren't time-sensitive so it shouldn't be big deal waiting a little longer or using a little more compute.\n\nThe paper has a lot of equations, but it boils down to a fairly simple method using multi-head cross-attention. I appreciated the detail, but I think the paper could be greatly simplified.\n\nThe related work section lists a lot of papers, but doesn't provide much of a comparison to the proposed method.\n\nWhat is the intuition of \"folding\" in XABD? I think this was a key part of the paper, but it wasn't motivated. Similarly, what is the intuition for ABLA?\n\nMinor: \n\nWhat are \"attributes\" in XABA? I found this nomenclature confusing.\n\nFigures should be vectorized.\n\nI would move the \"Attention Mechanisms\" subsection to an appendix (aside from explaining what MAB is).\n\nTables and Figures could be organized a little bit better. Hard to read in the text when it's flowing in every direction around the figures and tables.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster