Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning
Authors: Jannik Kossen
Neil Band
Clare Lyle
Aidan Gomez
Tom Rainforth
Yarin Gal
Venue: NeurIPS 2021 Poster (2021)
Keywords: attention
self-attention
transformers
multi-head self-attention
dot-product attention
equivariant
equivariance
invariant
invariance
interactions
tabular
supervised learning
masking

Abstract: We challenge a common assumption underlying most supervised deep learning: that a model makes a prediction depending only on its parameters and the features of a single input. To this end, we introduce a general-purpose deep learning architecture that takes as input the entire dataset instead of processing one datapoint at a time. Our approach uses self-attention to reason about relationships between datapoints explicitly, which can be seen as realizing non-parametric models using parametric attention mechanisms. However, unlike conventional non-parametric models, we let the model learn end-to-end from the data how to make use of other datapoints for prediction. Empirically, our models solve cross-datapoint lookup and complex reasoning tasks unsolvable by traditional deep learning models. We show highly competitive results on tabular data, early results on CIFAR-10, and give insight into how the model makes use of the interactions between points.


Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning
Jannik Kossen
Neil Band
Clare Lyle" Aidan N. Gomez 1.3 Tom Rainforth Yarin Gal
OATML, Department of Computer Science, University of Oxford 2 Department of Statistics, University of Oxford
Cohere
Abstract
We challenge a common assumption underlying most supervised deep learning: that model makes prediction depending only its parameters and the features a single input. To this end, we introduce a general-purpot architecture that takes input the entire dataser instead of processing one datapoint at a time. Our approach uses self-attention to reason about relationships between datapoints explicitly, which can be seen as realizing non-pan non-parametric models using parametric attention mechanisms. However, unlike conventional pon-parametrrc models, let the model learn end-to-end from the data how to make use of other datapoints for prediction. Empirically, our models solve cross-datapoint lookup and complex reasoning tasks vnsolvable by traditional deep learning models, We show highly competitive results on tabular data, early results on CIFAR- 10, and give insight into how the model makes use of the interactions between points.
deep learning
1 Introduction
From CNNs [57] to Transformers [90], most of supervised deep learning relies on parametric modeling: models learn parameters 0 from a set of training data Drain (1),,,))..... (æn, yn ) to maximize training likelihoods p(y æ; 0) mapping from features x X to target values y E ). At test time, they then make a prediction p(y" x=: that depends only those parameters 0 and the test input x". That is, parametric models do not consider direct dependencies between datapoints.
This paper challenges parametric modeling as the dominant paradigm in deep learning. Based on the same end-to-ent learning motivations that underpin deep learning itself, we consider giving models the additional flexibility of using training data directly when making predictions p(y" ax"" Dumin O).
Concretely, we introduce Nor-marrmt Transformers (NPTs): general deep learning architec- ture that takes the entire dataset as input and predicts by explicitly learning interactions between datapoints (Fig. 1). NPTs leverage both parametric and non-parametric predictive mechanisms, with the use of end-to-end training allowing the model to naturally learn from the data how to balance the two. Namely, instead of just learning predictive functions from the features the targets of independent datapoints, NPTs can also learn to reason about general relationships between inputs. We use head self-attention [4. 59, 90] to model relationships between datapoints and construct
Equal Contribution. Correspondence kossen, neil.band) @ @cs.ox.ac.uk.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Attributes
X;
Features
?
Datapoints
X M
Target
X [.j
Entry
(a) Input Data
(b)
Notation
(c) Parametric Model
(d) NPT
Figure I: NPTs learn direct interactions between datapoints. (a) Input data: predict masked target
entry [?] for datapoint X, (b) Notation from $2. (c) Parametric models predict only from the features
of the given input. (d) NPTs predict by modeling relationships between all points in the dataset.
a training objective for NPTs with a stochastic masking mechanism inspired by self-supervised reconstruction tasks in natural language processing [24]. We show that these models learn look up information from other datapoints and capture the causal mechanism generating the data in semi-synthetic settings. However, unlike conventional non-parametric models, NPTs are not forced to only make predictions in this way: they can also use the power of ordinary parametric deep learning.
Background. While questioning parametric modeling assumptions is unconventional in deep learning, in statistics, so-called non-parametric models are well-known and long-establisher field of study. Non-parametra
models make predictions in explicit dependence of the training data p(y" x=" Dunin). The most popular example of such models in the machine learning community are perhaps Gaussian Processes [74]. Non-parametric models typically do not require any training of parameters, and instead often directly interpolate between training points according to fixed procedure, e.g., [74, p.17]. The interactions between inputs are fully defined by architectural choices and a small set of hyperparameters that must be carefully chosen. Conventional non-parametric models cannot learn - in the sense familiar to deep learning practitioners interactions from the data, limiting the flexibility these models have in adapting to the data hand. Approaches such as Deep Gaussian Processes [22], Deep Kernel Learning [95], and Neural Processes [36, 37, 49] have all sought to apply ideas from deep neural networks to non non-marrmesrics.. Compared NPTs, these approaches rely heavily on motivations from stochastic processes. This leads to them being either less flexible than NPTs or requiring strong assumptions on the data, making them inapplicable the practical scenarios considered in this paper (cf. $3). Unlike previous work, NPTs explicitly learn to predict from interactions between datapoints, and they can be applied to general supervised machine learning tasks. We refer to $3 for an overview of these and other related approaches.
A key contribution of this paper is opening the door more general treatment of how deep learning models can make use of dependencies between datapoints for predictions. Our results demonstrate that NPTs make use of interactions between datapoints in practice, and we show highly competitive performance on several established tabular datasets as well as early image classification results. Additionally, we show that NPTs can solve complex reasoning tasks by combining representation learning and cross cross-datapoin lookup; something that is impossible for conventional deep learning or non con-aaraettric models due to their inability learn relations between datapoints.
We next discuss the specifics of our model ($2), before moving on to related work ($3), empirical results ($4), and finally, limitations, future work, and conclusions ($55).
Non-Parametric Transformers
Non-Parametric Transformers (NPTs) explicitly learn relationships between datapoints improve predictions. To accomplish this, they rely on three main ingredients: (1) We provide the model with the entire dataset - all datapoints- as input. We approximate this with minibatches where necessary for large data. At test time, both training and test data are input to the model; during training, the model learns to predict targets from the training data ($2.6). (2) We use self-attention between datapoints to explicitly model relationships between datapoints. For example, at test time, the attention mechanism models relationships amongst training points, amongst test points, and between the two. (3) NPT's training objective is to reconstruct a corrupted version of the input dataset. Similar to BERT [24], we apply stochastic masking to inputs and minimize a loss on predictions at entries masked out in the input. Next, introduce the three components in detail.

---

Reshupe
Reshape
Repeal
(c) &
2-
(a) Input
(b) Embedding
(c) Datapoint Attention
(d) Attribute Attention
Figure 2: Overview of the Non-Parametric Transformer. (a) The input dataset and mask matrix stacked and (b) linearly embedded for all datapoints independently. NPT then applies (c) Attention
Between Datapoints (ABD, $2.4) across all n samples of hidden dimension h d e. (d) Attention
Between Attributes (ABA, $2.5) then attends between the attributes for each datapoint independently, We repeat steps (c) and (d) and obtain final prediction from separate linear projection (not shown).
2.1 Datasets as Inputs
NPTs take as input the entire dataset E Raadd The datapoints are stacked as the rows of this matrix {X,. E Rd i E 1 n.)., and we refer to the columns as attributes {X... E IR" 11..... l...dd. Each attribute is assumed to share a semantic meaning among all datapoints. In single-target classification and regression, we assume that the targets (labels) are the final attribute X ;,d. and the other attributes {X., j # d} are input features, c.g., the pixels of an image. Each Xi,, is an entry or value. In addition to tabular data, many modalities such images, graphs, timeseries this format. Note that this is departure from common notation for supervised learning as introduced in I, as the input X now includes both features and targets (collectively, attributes).
be reshaped to fit
In masked language modeling [24], mask tokens denote which words sentence are unknown and where, at training time, model predictions will have a loss backpropagated. Analogously, we use binary matrix M E Raxd to specify which entries are masked in the input X. This matrix is also passed to NPT as input. The task is to predict the masked values X
(Xi,) MMaj = 1} from X
the observed values X {X.,
Mi, 0}. i.e., to predict p( X M
In summary, NPT takes input the entire dataset and masking matrix (X, M). and makes predictions X E Inndd for values masked at input. This general setup accommodates many machine learning settings simply by adjusting the placement of the binary masks in M. We focus single-target classification and regression corresponding to masking matrix M with is at all entries of the label column x - but outine multi-target settings, imputation, supervvienn using input features, and semi- semi-supervision in Appendix C.4. Next, we describe the NPT architecture.
2.2 NPT Architecture
An overview of the -on-maramcttic Transformer (NPT) is depicted in Fig. NPT receives the dataset and masking matrix (X, , M) ) as input (Fig. 2a). We stack these and apply an identical linear embedding to each of n datapoints, obtaining an input representation (0) E Ruxdxe (Fig. 2b). Next, we apply sequence of multi-heas self-attention layers [4. 24, 90]. Crucially, we alternatingly apply attention between datapoints and attention between attributes of individual datapoints (Figs. 2c-d).
These operations allow our model to learn both relationships between datapoints as well trans- formations of individual datapoints. Finally, an output embedding gives the prediction X E R which now has predicted values entries that were masked at input. We refer Appendix C.3 for details, such as treatment of categorical and continuous variables. Importantly:
Property 1. NPTs are equivariant to a permutationn of the datapoints. (cf. Appendix A for proof.)
In other words, if the set of input datapoints shuffled, NPTs produce the same prediction but shuffled in an analogous manner. This explicitly encodes the assumption that the learned relations between datapoints should not depend on their ordering. At a high level, cormutation-equvarriinn because all components of NPTs are permutationnequivarian equivariant functions is itself permutation-equivaaiiii which plays an important role throughout the NPT architecture.
holds
and the composition of permutation- We now briefly recap multi-head self-attention
3

---

2.3 Multi-Head Self-Attention
Muat self- attention (MHSA) is a powerful mechanism for learning complex interactions between elements in an input sequence. Popularized in natural language processing [4, 24, 90], MHSA-bass models have since been successfully applied to many areas of machine learning (cf. $3).
Dot-product attention computes attention weights by comparing queries {Q: E IR1x [i.....)) with keys {K; E 1xx,, | i E J.....m. ultimately updating the representation of the queries by aggregating over values {V, E Rixh, | | 1... m) via the attention weights. We stack the queries. keys, and values into matrices Q E RRnke,, K Rmxhe, and V RRmx,, and, as is commonly done for convenience, assume h2- = he = h. Then, we compute dot-product attention as
Att( Q, K, V) softmax(QK" //hV
(1)
Multi-head dot-product attention concatenates a series of k independent attention heads
MHAtt(Q. K, V) = concat (O....., O2) W where axis=
(2)
Oj = Au( KW 5: VWy).
(3)
We learn embedding matrices W W wy E R jj E (1,...; R.} for each head j. and W E Raxh mixes outputs from different heads. Here, we focus on multi-head self self-attention, MH
MHAu((Q H, K H, V H), which uses the same inputs for queries, keys, and values. Following Transformer best practices to improve performance [16, 24, 59, 66, 90], we first add a residual branch and apply Layer Normalization (LN) [3] followed by MHSelfAll())
Res H HW + MHSclfAt(LN((H))
(4)
with learnable weight matrix W E R Then, we add another residual branch with LN and row-wise feed-forward network (rFF), finally giving the full multi-head self-attention layer as
MHSA(H)) = Res [f) + +FF(LN((es(HH)) E m
(5)
2.4 Attention Between Datapoints (ABD)
The Attention Between Datapoints (ABD) layer is key operation for NPT. It explicitly transforms
data by reasoning about pairwise relationships between all datapoints, see Fig. 2c. As input to ABD, we flatten the output of the previous layer (4) from Raxdxe to Ra with h = d e. Then, we
apply MHSA(.) between the intermediate datapoint representations (H
(8) E Rixh h | E (......) m) as
ABD(H -MHAA H (9)
H (B+1) E Ra
At the first ABD layer, we input (0)) xadxe the linearly embedded input data, After applying ABD, we reshape the output again. from R to R=dxxe Here, the rFF of each ABD layer is an MLP that applied independently to each of the n datapoints.
Note that this is distinct from how MHSA(.) is usually applied in the literature, as we compute attention between different datapoints and not between the features of a single datapoint [24, 25, 46, 90]. For example, in natural language processing, attention is usually applied between the tokens (attributes) of sentence (datapoint) but not between different sentences. For example, NPT could learn to attend between two datapoints with indices i and i' by embedding Q; and K; in close proximity. Following (1), datapoint will then attend more closely to i' because K,K? will be large. By stacking many ABD layers, NPT can learn higher-order interactions between datapoints [24, 90].
2.5 Attention Between Attributes (ABA)
We now introduce Attention Between Attributes (ABA), which by default perform after each ABD layer. ABA layers can help the model learn better per-datapoint representations for the between-datapoint interactions, see Fig. 2d. For ABA, apply MHSA(-) independently to each row (corresponding to a single datapoint) in the input (2) E [Rdxe, i (1,...,n), giving
ABA(H ( ())
stack (MHSA H (0) (....H))((H(())) = H (++1)_ E Raxxxe axis
(7)

---

Just like in standard Transformers [24, 25, 46, 90]. ABA is used to transform attribute representations of single datapoints independently. We batch over the dimension to compute ABA efficiently. By alternating between attention between datapoints (ABD) and attributes (ABA), NPT's can model both complex dependencies between points as well as learn suitable transformations of datapoints individually. Next, we describe the use of masking mechanisms during NPT training and evaluation.
2.6 Masking and Optimization
Masking. Much like in masked language modeling [24], we use masks to indicate which values NPT is expected to predict, and to prevent the model from accessing ground truth values. Recall that NPT needs to predict p(X x", with masked values X XM {X.,, M.,j = 1} and observed values X {x. M., = 0}. Masked values can be either features or targets. Canonically, masked language modeling is used to perform self-supervised learning on sequence of tokens in a sentence [24]. We use such stochastic feature masking to mask feature values X.,, j # d, with probability Pteature during training. We also apply stochastic masking to the targets of the training set .. with probability Pharget- We call this stochastic targel masking. Note that we take great care to avoid test set leakage and never reveal targets of the test set to NPT. We refer to Appendix C.4 for full details of our masking procedure in a variety of settings.
NPT Objective. During training, we compute the negative log-likelihood loss at training targets [Trgets well the auxiliary loss from masked-out features C We write the NPT training objective as NPT (1 - Tangess + FeFeatures where À is hyperparameter. At test time, we only mask and compute a loss over the targets of test points. See Appendix C.5 for optimization details.
This objective has a few notable elements. Feature masking requires NPTs to make predictions over all attributes, encouraging the models to learn representation of the entire dataset. This increases the difficulty of the task and adds more supervision, which we find tends to have a beneficial regularizing effect. Interestingly, stochastic larget masking means that many training targets are unmasked the model at training time. This allows NPT's to learn to predict the masked targets of certain training datapoints using the targets of other training datapoints in addition all input features. ? NPTs no longer have to memorize a mapping between training inputs and outputs in their parameters 0, and can instead use their representational capacity learn functions using other training features and targets as impur. For example, NPTs could learn to assign test datapoints to clusters of training datapoints, and predict on those points using interpolation of the training targets their respective cluster. We explore the ability of NPTs to solve such tasks in $4.2. Further, we study more complex extensions to these tasks, which cannot be solved by simple interpolative models, in Appendix B. 1.2
Handling Large Datasets. Due to the poor O(n2) time and space complexity of self-attention, we resort approximations once the data grows too large. For example, we reach 24 GB of GPU memory for standard NPT model sizes at about 8000 datapoints. We find that processing the data in random subsets for model training and prediction, i.e., minibatching, is simple and effective solution. We construct minibatches such that, at test time, training and test data are both present in the same batch, to allow NPTs attend to training datapoints. In $4.3, we show that NPTs make use of attention between datapoints with minibatching enabled. See $5 for further discussion and ideas for future work.
3 Related Work
Deep Non-Parametric Models. Deep Gaussian Processes [22] and Deep Kernel Learning (DKL) [95] extend ideas from Gaussian Processes [74] to representation learning. Deep GPs stack standard GPs with the aim to learn more expressive relationships between input points, sharing motivation with NPTs. However, unlike NPTs, deep GPs are difficult to work with in practice, requiring complex approximate inference schemes [13, 21, 77]. DKL applies a neural network to each datapoint independently before passing points on to standard Gaussian Process, making predictions based directly on similarity in embedding space instead of learning the interactions themselves.
Neural Processes. Similar to GPs, Neural Processes (NPs) [36, 37] define distribution over functions. They use a latent variable model parametrized by neural networks, fulfilling specific
A concern here could be that the model will memorize training targets and fail to generalize. In practice, we do not observe generalization issues, likely because (i) a loss is never backpropagated on an unmasked value, and (ii) BERT-style masking [24] uses token randomization prevent memorization. See Appendix C.4.
5

---

architectural constraints to approximately preserve consistency of amite-dimensionas marginals. Attentive Neural Processes (ANPs) [49] extend Neural Processes to allow for direct attention between a context set and targets. However, as the authors themselves stress, "NPs and GPs have different training regimes" [49]. While GP can be trained on a single dataset, (A)NPs require multiple realizations of the dataset. The authors further note that "a direct comparison between the two is usually not plausible" " [49], which is why we cannot compare (A)NPs to NPTs on our standard tasks.
Attention. NPTs are part of a line of recent work that explores the use of Transformer-based architectures outside of natural language processing, e.g.. Transformers in computer vision [25, 46, 67] or architectures exploiting desirable invariances or equivariances [33, 44, 59, 61]. Like NPTs, Set Transformer [59] attends to set of imput points. However, unlike NPTs, Set Transformer relies on the existence of multiple independent sets for training and makes only single prediction for each set, Like NPTs, Axial Transformers [42] and MSA Transformers [73] attend to multiple dimensions of matrix-shaped input. However, Axial Transformers process single images as input, i.e., no attention across datapoints is performed. MSA Transformers use attention within individual protein sequences and across an aligned protein family for contact prediction, but do not consider more general setting. Recent works have improved neural network performance on tabular data using attention. AutoInt [80] is direct application of multi-head attention to tabular data, and TabNet [2] sequentially attends to sparse subsets of the features inspired by tree- base models. Both approaches do not reason about interactions between datapoints, key contribution that we introduce with NPT in this work.
Few-Shot Learning, Leeta-Lernning.. and Prompting. In $4.2, we apply NPTs to tasks that require learning of relational structure between datapoints on training data to achieve good generalization performance on novel test inputs. This setup shares motivations with meta-learning [6, 8, 29, 56), in which a model is pre-trained on variety of tasks, such that it can then learn new tasks using only a small number of additional training points from the new task. However, we consider evaluation without any additional gradient updates, unlike recent meta-learning methods [29, 97] which are therefore inapplicable to this setting. Recent works on few-shot learning with text prompting [12, 72] provide a trained Transformer-basee language model with few examples of novel relationship in prompt at prediction time, where they observe strong generalization on the task. Similarly, we consider attention between "context" of datapoints. While ground-trute input-output pairs provided for prompting, we consider settings in which no ground-truth is given at prediction time (cf. Appendix B.1.2), but the model can solve the task if it has learned the underlying relational structure.
Sepi-Supe L Learning and Graph Neural Networks. NPTs relate to work on cup-suueee learning [15, 27, 51] and transductive learning [89], which both make use of unlabeled inputs during training. NPTs natively support this by simply including any unlabeled datapoints with masked-out targets in the input matrix at training time. This body of related work includes semi-supervised and transductive learning on graphs using graph neural networks (GNNs), e.g., [34, 52, 53, 91, 96).
NPTs can be seen a generalization of GNNs in which a set of dependencies (edges) between datapoints is not known a priori and is instead learned from data using self-attention. Like NPTs, Neural Relational Inference (NRI) [53] attempts to discover relations amongst datapoints. However, NRI lacks scalability because it requires that embeddings be stored for each potential graph edge.
Metric Learning. (Deep) Metric Learning aims to learn distance functions such that the (semantic) similarity and dissimilarity between input points is meaningfully captured, e-g., [65, 76, 79, 92-94]. Similarly, retrieval models in NLP learn to look up relevant training instances for prediction [38, 39, 41]. The attention between datapoints in NPTs can be seen as implicitly learning exactly such (dis-)similarity. Usually, metric learning embeds inputs by applying the same embedding function independently to each datapoint. This is in contrast to NPTs, which leverage a learned self-ateention mechanism between test inputs and training datapoints (including their labels) at prediction time.
4 Experiments
We seek to answer the following set of questions in our evaluation of NPT's: (Q1) How do NPTs perform on standard benchmarks for supervised machine learning? (Q2) Can NPTs successfully model interactions between datapoints in idealized settings? (Q3) Do NPTs actually learn to rely on interactions between datapoints for prediction on real-world datasets? (Q4) If so, what is the nature of these interactions, e.g., which other datapoints are relevant for prediction?
We release code for NPTs github.com/OATTMLASS CarrroAaAMMMLNNoooaaaeiiiiiiii www.aaettri---Taasfmeeeee

---

Table 1: Average rank order of various methods (= standard error) on UCI benchmarks, across binary classification, multi-class classification, and regression tasks. We determine rank using the test area under the receiver operating characteristic (AUROC) curve on binary classification (4 of 10 datasets), accuracy on multi-class classification (2 of 10), and root mean squared error (RMSE) on regression (4 of 10), and sort methods by ascending rank for each metric. See Appendix B.7 for the full results.
Method
AUROC
Method
Accuracy
Method
RMSE
NPT
2.50 + 0.87
NPT
2.50 + 0.50
CatBoost
3.00 + 0.91
CatBoost
2.75 0.85
XGBoost
2.50 - 1.50
XGBoost
3.25 0.63
LightGBM
3,50 = 1.55
MLP
3,00 = 2.00
NPT
3.25 + 1.31
XGBoost
4.75 + 1.25
CatBoost
3.50 - 0.50
Gradient Boosting
4.00 1 1.08
Gradient Boosting
5.00 0.71
Gradient Boosting
3.50 1 1.50
Random Forest
4,50 11: 0.87
MLP
5.75 1.49
Random Forest
6.50 0.50
MLP
5.00 1.22
Random Forest
6.00 - 0.71
TabNet
7.50 0.50
LightGBM
6.50 + 1.55
TabNet
6.50 + 1.32
LighIGBM
7.50 + 1.50
TabNet
6.75 + 0.95
K-NN
8.25 0.48
k-NN
8.50 0.50
k-NN
8.75 0.25
4.1 NPTs Perform Competitively on Established Benchmarks
To answer (Q1), we evaluate NPTs on tabular data from the UCI Repository [26] as well as the CIFAR- [55] and MNIST [58] image classification datasets. Tabular data ubiquitous in real-world machine learning [20] but notoriously challenging for general purpose deep neural networks, which
are rarely used in practice here because they are consistently outperformed by boosting models [78].*
Tabular Datasets, Setup, and Baselines. We evaluate NPTs over 10 datasets varying across the
number of datapoints, number of features, composition (categorical or continuous) of features, and task. 4 of the 10 are binary classification, 2 are multi-class classification, and 4 are regression. We compare NPT against a wide set of standard state-of-the-ar baselines: Random Forests [10], Gradient Boosting Trees [32], XGBoost [17], CatBoost [71], LightGBM [48], MLPs, k-NN [1, 30), and TabNet [2]. For additional background on tree-based models, see Appendix D.I. We tune the parameters of all models validation sets and use 10-fold cross- validatili whenever computationally feasible. Note that while we perform an extensive grid search for the baselines, we only search over a small set of configurations for NPTs. We refer the reader to Appendix E for further details on the setup for datasets and baselines, and Appendix C.I for NPT hyperparameters.
Tabular Data Results. We report the average rank order for NPT and various tree-based and deep learning baselines in Table NPT achieves the highest average ranking on binary and multi-class classification tasks, outperforming CatBoost and XGBoost, two popular state-of-the-art boosting methods designed specifically for tabular data. On regression tasks, NPT ties in average rank with XGBoost, and is outperformed only by CatBoost, In addition to its strong rank-wise performance, NPT achieves best performance on 4 of the 10 benchmark datasets more than any other method. We find that these are remarkable results for a general purpose model that does not include subular-specinf design, supporting our hypothesis that attention between datapoints a useful architectural inductive bias for prediction. For all metrics across all datasets, i.e., NL. for classification, AUROC/accuracy for binary/multi-clan class classification, and (R) for regression, we refer the reader to Appendix B.7. In the appendix, we present ablations which suggest that the performance of NPT is robust across wide range of hyperparameter choices (Appendix B.4) and that both the introduction of the ABA layer and the stochastic feature masking contribute positively to the performance of NPTs (Appendix B.5).
Image Data Results. On CIF we replace our linear encoder with a CNN followed by ABD layers on the CNN encodings, achieving test accuracy of 93. 73.7% We achieve 98.3% accuracy on MNIST using linear patching
[25]. Crucially, we show in $4.3 that NPTs learn make use of interactions between images on both the CIFAR- 10 and MNIST datasets, supporting the claim that attention between datapoints is useful beyond tabular data. We also explore linear patching on CIFAR-10. See Appendix B.8 for these results along with setup details and further discussion.
*We conduct an informal survey of all Kaggle [45] competitions using tabular data completed in 2020 with a public leaderboard. In 11 out of total of 13 cases, the winning entries relied on some form of boosting.
7

---

Original Masked Targets
Duplicate Original
Predicted Target
99.9%
Sample
Duplicate
No Masking
X1,4
Can look
attention
performs
up
lookup
X2,4
target
value
Xa.4
Original Duplicate
Duplicate Target
(a) Semi-Synthetic Input
(b) Attention Weights
(c) Model Predictions
Datapoint
Datgpoint
Datapoint
2
99 585%
r 99.9615
99.98%
Predicted Target
Ditapoint 4
Datopoint
Avg. of 1-9146
2
99 Syr
na 99 9599
99 6o..666-3. 666
Intervention Duplicate Target
(d) Interventions on Duplicates
(e) Model Responses to Interventions
Figure 3: Demonstrating NPT's ability predict from Attention Between Datapoints (ABD). (a) We append to the original data with masked targets [?] copy of the same data with all masked values revealed, such that perfect prediction via lookup is possible. (b) Attention weights indicate that the ideal lookup behavior is learned by NPT. Shown are actual values learned by NPT at head 0 and depth 4 for the first 3 datapoints, we intervene on the values of individual targets, (e) finding that NPT predictions adjust accordingly.
(c) NPT predictions closely match the ideal values. (d) Additionally,
4.2 NPTs Can Learn to Predict Using Attention Between Datapoints
To determine if NPTs can successfully learn to exploit interactions between datapoints (Q2), we intro- duce task with strong input correlations for which know ground-truth interactions. Concretely, we use the UCI Protein regression dataset (cf. $4.1) to construct the following semi-synthetic task: for each batch, we input the original data with masked target values well as copy of the original data where all target values have been revealed, i.e., no masking is applied (Fig. 3a). NPTs can use attention between datapoints achieve arbitrarily good performance by learning look up the target values in the matching duplicate row. At test time, we input novel semi-synthetic test data to ensure that NPT has learned the correct relational mechanism and not just memorized target values.
NPTs successfully learn to perform this lookup between original and duplicate datapoints. The ABD attention weights, visualized for the first three datapoints in Fig. 3b, clearly show the model correctly attending to the duplicates. As a result, NPT predictions are Pearson-correlater with the duplicate targets at 99.9% (Fig. 3c). This equals RMSE of only 0.44, about magnitude lower than the error on the original Protein dataset (Table 11). We conclude that NPTs learn to predict by looking up the target values from matching points. Further discussion and attention maps are in Appendix 1.1.1.
Purely parametric models cannot exploit information from other datapoints, limiting their performance. For example, MIPs achieve an RMSE of 3.62 on this task. Non-parametre approaches also cannot solve this task in its original form, because unlike NPTs they must be told which datapoints are the originals (training data) and which the duplicates (test data) well as which columns contain features and which target values. We demonstrate in Appendix 1.2 that even when we make these concessions, we can easily adapt the task such that both k-Nearest Neighbors and Deep Kernel Learning fail to solve it. In fact, we are not aware of any other model that can solve the adapted task.
Additionally, we perform an interventional experiment to investigate the extent to which NPTs have actually learned the causal mechanism underlying the lookup task. As illustrated in Fig. 3d, we now intervene on individual duplicate datapoints test time by varying their target value across wide range. We stress that we perform these experiments without retraining the model, using exactly the same NPT from Figs. 3a-c. The model is now confronted with target values associated with features
8

---

Table Drop in NPT performance after destroying information from other datapoints. Shown are changes in test set performance, where negative values indicate worse performance after corruption.
Accuracy
CIFAR-IE 10
Poker
Income
Higgs
MNIST
Forest
Kick
Breast Cancer
-1.2
-1.1
-1.1
-0.5
-0.4
-0.1
0.0
ARMSt/RMSE (%)
Yacht
Protein
Boston
Concrete
- 52%
-21%
-20%
-7%
that are highly unlikely under the training data. This label distribution shift [35] challenging setting for neural networks. However, NPT predictions follow the intervened target values with near-perfect correlation, Fig. 3e, continuing to predict by correctly looking up targets.
We now confidently conclude that NPTs robustly learn the causal data generating mechanism under- lying the semi-synthetic dataset. This requires NPTs to learn non-trivial sequence of compuational steps. They must learn to match rows based on similarity of relevant features; to look up the target value of the duplicated datapoint; and, copy that value into the target of the masked datapoint.
4.3 NPTs Learn to Use Attention Between Datapoints on Real Data
We next consider (Q3): do NPTs actually learn to use attention between datapoints for prediction on real data? We design a test that allows NPT trained in standard fashion on one of our benchmark datasets depend on relationships between datapoints at test time. Concretely, for each target value in the input we randomize the data for all other datapoints by independently shuffling each of their attributes across the rows. We then evaluate the loss on the prediction at the target entry and repeat this procedure for all test datapoints. This completely corrupts the information from all datapoints except the one for which we evaluate. Hence, a model that relies meaningfully on attention between datapoints will show deteriorating performance. We give an algorithm for the corruption procedure as well as further discussion in Appendix B.2.1.
quantify the extent to which the predictions of an
We report the resulting change in performance after corruption in Table 2 for all datasets from $4.1. We find that for most datasets, the corruption of other rows at test time significantly decreases the performance of the trained NPT models. This indicates that the NPTs have successfully learned to make predictions supported by attention between datapoints. For some datasets, the corruption experiment deteriorates performance completely. For example, for the Protein regression dataset NPT achieves Mlate-of-the-a performance, but corrupting the input at test time leads to NPT performing worse than all of the baselines considered in 1. We note that minor differences in performance often still significant, as differences between competing models in $4.1 are often likewise small.
Interestingly, certain datasets such Forest Cover, Kick, and Breast Cancer, corrupted inputs do not significantly affect performance. It appears that when NPTs do not find it advantageous to rely attention between datapoints during training, they learn completely ignore other inputs, essentially collapsing into a standard parametric model. This supports our earlier claims that NPTs learn end-to-end from data the extent to which they rely other datapoints for prediction. We think this is extremely interesting behavior and are unaware of prior work reporting similar results. However, we stress that these results reflect inductive biases of the NPT architecture and do not lend themselves to general statements about the performance of parametric versus non-parametric models.
4.4 NPTs Rely on Similar Datapoints for Predictions on Real Data
So far, we have presented convincing evidence that NPTs (sometimes strongly) depend on attention between datapoints. However, we do not know what kind of interactions are learned in practice on real data (Q4). As an initial step towards understanding this, we present two experiments investigating to which other datapoints NPT attends.
Qualitative Evidence. Figure 4 shows an attention map for attention between datapoints (ABD) of NPT on batch of the Protein regression dataset. We sort the input data with respect to their input space distance such that similar datapoints are now close to each other. The diagonal pattern

---

in Fig. 4 indicates that NPT attends more strongly to datapoints that are similar in feature space. Appendix B.3.1 discusses this further and gives additional attention maps.
Quantitative Evidence. Seeking a quantitative measure for this hy- pothesis, the data deletion experiment repeats the following procedure for all test set points: iteratively delete other datapoints from the input if they do not significantly affect the prediction. We stop if less than 2% of the original datapoints remain, or if the total change in prediction for the target (relative to the original prediction with all data) exceeds 10%. We investigate the average input feature space distances between the test point and the kept datapoints, as well the distances between the test point and the deleted datapoints. "Input features" here refer to all attributes of the input datapoints that are not labels.
Attending From
103
10²
Attending To
We find that kept datapoints have a significantly lower average feature space distance the test point than those deleted. This indicates that two datapoints i, i' that are similar in input feature space, such that [x, )² is low, have larger effect the predictions of one another. Wilcoxon signed-rank test is significant p 8.77 10-130 We give full details this in Appendix B.3.2.
Fig. 4: Attention weights.
Both experiments support the hypothesis that NPTs rely on similar datapoints for prediction real data settings. One possible explanation is that similar datapoints might have different realizations of observation noise which NPTs could learn to average out. Altogether, we conclude that NPTs can and do learn representations which rely interactions between datapoints for prediction.
5 Limitations, Future Work, and Conclusions
Limitations. NPTs share scaling limitations with all naïvely con-parametric approaches [74] and GNNs [52]. We demonstrate this in a preliminary analysis of the computational cost of NPTs and the baseline methods - including training time and CPU/GPU memory requirements - in Appendix B.6. While we have seen success with random minibatching ($2.6), future work might consider applying principled attention approximations, such as learning representative input points [59], kernelization [19, 47], or other apprrity-induccng methods [5, 18, 84], improve the scalability of NPTs.
Future Work. We believe that the unique predictive mechanism of NPTs makes them an inter- esting object of study for other tasks including continual learning, multi-task learning, few-shot generalization, and domain adaptation. For example, when predicting under distribution shift, general relations between datapoints and attributes may remain valid and allow NPTs to accommodate such scenarios better. Additionally, future work could explore the connections to stochastic processes, e.g.. by extending NPTs to be approximately consistent, similar to Neural Processes [36, 37, 49].
Conclusions. We have introduced Non-Parametric Transformers (NPTs), novel deep learning architecture that takes the entire dataset input and uses self-attention model complex relationships between datapoints. NPTs challenge and naturally extend parametric modeling as the dominant paradigm deep learning. They have the additional flexibility to learn to predict by directly attending to other datapoints. Notably, NPTs learn this end-to-end from the data at hand. Empirically, NPTs achieve highly competitive performance on variety of benchmarks, and additional experiments demonstrate their ability to solve complex reasoning tasks over datapoints. Further, we show that on real data, NPTs learn to rely on attention between datapoints for prediction. We believe that the characteristics of NPTs will make them exciting object of further study.
Acknowledgments and Disclosure of Funding
We acknowledge funding from the New College Yeotown Scholarship (JK), the Rhodes Trust (NB), and the Open Philanthropy AI Fellowship (CL). We thank Lewis Smith, Pascal Notin, Uri Shalit, Joost van Amersfoort, Sören Mindermann, Lood van Niekerk, and the anonymous reviewers for helpful feedback and interesting discussions that have led numerous improvements of the paper.
10

---

References
[1] Naomi S Altman. An introduction keree and neirest-neighhor nonparametric regression. The American Statistician, 46, 1992.
Sercan O Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. arXiv:1908.07442, 2019.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoftrey Hinton. Layer normalization. arXiv: 2016.
[4] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations. 2015.
[5] Iz Beltagy, Matthew E. Peters, and Arman Cohan. arXiv:2004.05155 2020.
Longformer:
The long-document transformer.
[6] Y. Bengio, Bengio, and J. Cloutier. Learning synaptic learning rule. International Joint Conference on Neural Nenworks, volume 2, 1991.
[7] L. Bentley. Multidimensional binary search trees used associative searching. In Communications of the ACM, volume 18, 1975.
[8] John B Biggs. The role of metalcarning study processes. British journal of educational psychology, 55, 1985.
[9] Leo Breiman. Bagging predictors. Machine learning, 24, 1996.
[10] Leo Breiman. Random forests. Machine learning. 45, 2001.
[11] Leo Breiman, Jerome Friedman, Charles Stone, and Richard A Olshen, Classification and regression trees. CRC press, 1984.
[12] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamers.. arXiv:2005. 14165, 2020.
[13] Thang Bui, Daniel Hernández Lobato, Jose Sernandez-Lobato Yingzhen Li, and Richard Turner. Deep gaussian processes for regression using approximate expectation propagation. In International Conference Machine Learning, 2016.
[14] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voo Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. arXiv:2012.07805, 2020.
[15] Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning. IEFF Transactions on Neural Networks, 20(3):542-542, 2009.
[16] Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Combining recent advances in neural machine translation. In Annual Meeting of the Association for Computational Linguistics, volume 56, 2018.
[17] Tianqi Chen and Carlos Guestrin. Xgboost: scalable tree boosting system. In Knowledge Discovery and Data Mining, volume 22, 2016.
[18] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv:1904.10500 2019.
[19] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021.
[20] Michael Chui, James Manyika, Mehdi Miremadi, Nicolaus Henke, Rita Chung, Picter Nel, and Sankalp Malhotra, Notes from the AI frontier: Insights from hundreds of use cases, 2018.
[21] Zhenwen Dai, Andreas Damianou, Javier González, and Neil Lawrence. Variational auto auto-encoded deep gaussian processes. In International Conference on Learning Representations, 2016.
11

---

[22] Andreas Damianou and Neil D Lawrence. Deep gaussian processes. In International Conference on Artificial Intelligence and Statistics, volume 16, 2013.
[23] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image datahase. In Conference on Computer Vision and Pattern Recognition, 2009.
[24] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding, arXiv: 2018.
[25] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[26] Dhecru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://///arc uci. edu/ml.
[27] Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In International Conference on Artificial Intelligence and Statistics, volume 13, pages 201-208, 2010.
[28] Angelos Filos, Sebastian Farquhar, Aidan N Gomez, Tim GJ Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Amoud De Kroon, and Yarin Gal. A systematic comparison of bayesian deep learning robustness in diabetic retinopathy tasks. In NeurlPS Workshop on Bayesian Deep Learning, 2019.
[29] Chelsea Finn, Pieter Abbcel, and Sergey Levine, Model-agnostic meta- meta-lcarming for fast adaptation of deep networks. In International Conference on Machine Learning, volume 34, 2017.
[30] Evelyn Fix. Discriminatory analysix: nonparametric discrimination, consistency properties, volume 1. USAF school of Aviation Medicine, 1985.
[31] Yoav Freund and Robert Schapire. A decision-theoretic generalization of on-line learning and an application to hoosting. Journal of computer and system sciences, 55, 1997.
[32] Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics. 2001.
[33] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. te(n)-transsormer:: 3d roto-translation equivariant attention networks. In Advances in Neural Information Processing Systems, volume 33, 2020.
[34] Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. In International Conference on Learning Representations, 2018
[35] Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. In Advances in Neural Information Processing Systems, volume 33, 2020.
[36] Marta Gamelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Tch, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, volume 35, 2018.
[37] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv: 107.01627.0 2018
[38] Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences hy editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450, 2018.
[39] Kelvin Guu, Kenton Lec, Zora Tung, Panupong Pasupat, and Ming-Wey Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv:2002.08900 08900, 2020.
[40] Charles R. Harris, K. Jarrod Millman, Stéfan van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor. Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Kérand-Marchan Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy, Nature, 585, 2020.
[41] Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy Liang. A retrieve-and-cdii framework for predicting structured outputs. In Advances in neural information precessing systems, 2018.
[42] Jonathan Ho. Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. 1rXi::111..12180, 2019.
12

---

[43] James Honaker and Gary King. What to do about missing values in time series cross-section data. American Journal of Political Science, 2010.
[44] Michael Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee Whye Teh, and Hyunjik Kim. Lietransformer: Equivariant self-attention for groups. arXiv:2012.1088 2020.
[45]
Google Inc. Kaggle. htrps://www.kaggle.cm. 2021.
[46] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention. arXiv:2103.03266 2021.
[47] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with lincar attention. In International Conference on Machine Learning, volume 37, 2020.
[48] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Yan Liu. Lightgdm: A highly efficient gradient boosting decision tree. In Advances in neural information processing systems, volume 30, 2017.
[49] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals. and Yee Whye Tch. Attentive neural processes. In International Conference on Learning Representations, 2019.
[50] Gary King, James Honaker, Anne Joseph, and Kenneth Scheve. Analyzing incomplete political science data: An alternative algorithm for multiple imputation. American Political Science Review, 2001.
[51] Diederik P Kingma, Danilo J Rezende, Shakir Mohamed, and Max Welling. Semi-supervised learning with deep generative models. arXiv:1406.5298, 2014.
[52] Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations. 2017.
[53] Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning, volume 35, 2018.
[54] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European Conference on Computer Vision, 2020.
[55] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
[56] Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350, 2015.
[S7] Y. LeCun, L. Bottou, Y. Bengio, and Haffner: Gradient-based learning applied document recognition. Proceedings of the IEEE, 86, 1998.
[58] Yann LeCum, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. 2010.
[59] Juho Lee, Yoonho Lee. Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutationninvariaa neural networks. In International Conference on Machine Learning, volume 36, 2019.
[60] T. Liu, A. Moore, and A. Gray. New algorithms for efficient high-dimensional nonparametric classification. In Journal of Machine Learning Research, volume 7, 2006.
[61] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dasovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, volume 33, 2020.
[62] Wei- Yin Loh. Fifty years of classification and regression trees. International Statistical Review, 82, 2014.
[63] Rhiannon Michelmore, Marta Kwiatkowska, and Yarin Gal. Evaluating uncertainty quantification in end-to-end autonomous driving control. arXiv:1811.000 108817, 2018.
[64] James N Morgan and John A Sonquist. Problems in the analysis of survey data, and a proposal. Journal of the American statistical association, 58, 1963.
13

---

[65] Yair Movshoviiz-Attias, Alexander Toshev. Thomas K Leung, Sergey loffe, and Saurabh Singh. No fuss distance metric learning using proxies. In International Conference Computer Vision, pages 360-368,
2017.
[66] Sharan Narang, Hyung Won Chung. Yi Tay, William Fedus, Thibault Févry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, and Colin Raffel. Do transformer modifications transfer across implementations and applications? 1rXiv:2102.11972. 2021.
[67] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, volume 35, 2018.
[68] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, ligh-performance deep learning library. In Fox, and R. Garnett, editors, Advances in
H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Bou Neural Information Processing Systems, volume 32, 2019.
[69] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2011.
[70] Google Cloud AT Platform. Getting started with the built-in tabnet algorithm, 2021. URL cloud. google. com/ ai plppppaafoaare aatafform/ae wtt.ttrrnnnnnntoet docs///////////////// algor i thms saab-net-ae start.
[71] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, 2018
[72] Alee Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amadei, and Nya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[73] Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier, John F Canny, Picter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. bioRxiv, 2021.
[74] Carl Edward Rasmussen, Gaussian processes in machine learning, In Summer school on machine learning. 2003.
[75] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Mancd Imagenet-21k pretraining for the masses. arXiv:2104.1 10972, 2021.
[76] Sam Roweis, Geoffrey Hinton, and Ruslan Salakhutdinov. Neighbourhood component analysis. In Advances in Neural Information Processing Systems, volume 17, page 4, 2004.
[77] Hugh Salimbeni and Mare Deisenroth. Doubly stochastic variational inference for deep gaussian processes. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, 2017.
[78] Robert Schapire. The strength weak learnability, Machine learning. 5, 1990.
[79] Jenny Seidenschwarz, Ismail Elezi, and Laura Leal-Taixé. Learning intra-batch connections for deep metric learning. In International Conference on Machine Learning, 2021.
[80] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceedings of the 28th ACM International Conference Information and Knowledge Management, 2019.
[81] D.J. Stekhoven and P. Buchlmann. Missforest nonparametric missing value imputation for mixed-type data Bioinformatics, 2012.
[82] Yu-Sung Su, Andrew E. Gelman, Jennifer Hill, and Masanao Yajima. Multiple imputation with diagnostics (mi) in R: Opening windows into the black box. Journal of Statistical Saftware, 2012.
[83] C. Sun, A. Shrivastava. S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In 2017 IEEE International Conference Computer Vision (ICCV), 2017.
14

---

[84] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, arXiv:2009.0611 2020.
[85] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems, volume 32, 2019.
[86] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv:2012.12877, 2020.
[87] Betty Van Aken, Julian Risch, Ralf Krestel, and Alexander Löser. Challenges for toxic comment classifica- tion: An in-depth error analysis. arXiv preprint 80%.07200.. 2018.
[88] Stef van Buuren and Karin Groothuis-Oudshoomm mice: Multivariate imputation by chained equations Journal of Statistical Software. 2011.
[89] Vladimir Vapnik. Estimation of dependences based empirical data. Springer Science & Business Media, 2006.
[90] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakoh Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Information Processing Systems, volume 30, 2017.
[91] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
[92] Sethu Vijayakumar and Stefan Schaal. Local dimensionality reduction for locally weighted learning. In International Symposium on Computational Intelligence in Robotics and Automation, pages 220-225.
IEEE, 1997.
[93] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, volume 29, pages 3630-3638, 2016.
[94] Xinshao Wang, Yang Hua, Elyor Kodirov, Guosheng Hu, Romain Garnier, and Neil M Robertson. Ranked list loss for deep metric learning. In Conference Computer Vision and Pattern Recognition, pages 5207 5216, 2019.
[95] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. In International Conference on Artifciall Intelligence and Statistics, volume 19, 2016.
[96] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference Learning Representations. 2019.
[97] Jaesik Yoon, Taesup Kim, Ousmane Dia. Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 31, 2018
[98] Yang You, Jing Li. Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations, 2020.
[99] Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton. Lookahead optimizer: k steps forward, step back. In Advances in Neural Information Processing Systems, volume 32, 2019.
15


Reviews:
[{"review_id": "xKrH_ooJx-D", "reviewer": "Reviewer_m1rj", "summary": "This paper introduces a neural network architecture termed Non-Parametric Transformer (NPT) that uses self-attention (i) between data points and (ii) between features within each data point. The core conceptual contribution is to treat supervised deep learning problems as tasks where the model receives the entire dataset (incl. both labeled and unlabeled data points) as input — in contrast to the common approach of providing data samples independently to a deep learning model. The proposed NPT model is evaluated on tabular datasets where the authors report competitive results compared to boosted decision trees (which represent the state of the art). Applied on MNIST and CIFAR10, the NPT model performs reasonably well. The authors further perform a series of experiments to analyze the learned attention patterns and the importance of self-attention between data points for generalization performance.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a high-quality paper that is exceptionally well-written, well-structured and very clear in its exposition. The idea is novel and well-positioned against related work (with some exceptions). I think that this paper is relevant for the NeurIPS community and I expect that it will inspire a range of follow-up works.\n\nThe idea of using self-attention to take into account all other data points (including their labels, if available) both during training and inference is intriguing, and given the rising popularity Transformer-style architectures in the machine learning community, I expect that this method will spawn lots of interest despite its shortcomings in its current form, such as its computational complexity and its limited usefulness/applicability in any of the studied tasks in the paper.\n\nThe experimental evaluation is carried out with great care and the authors come up with several insightful experiments to study whether the attention between datapoint (ABD) mechanism learns to make use of features/labels of similar data points.\n\nOverall, I think that this paper can be accepted and that it would make a great addition to the conference program. I do, however, want to highlight several limitations of the method and issues with the paper below, which I encourage the authors to take into account in the next version of the paper.\n\nThe core contribution of the paper is the attention between datapoint (ABD) mechanism and its application for semi-supervised learning in tabular data. While the other model contribution, namely attention between attributes (ABA) is interesting, it is not properly experimentally evaluated (e.g. against an alternative, simpler architecture component, such as an MLP on the flattened representation, or simply the identity, i.e. no module at all). The reader might wonder why this module is introduced in the first place, and its necessity is unclear. I could imagine that this module could simply be left out for the tabular tasks without affecting performance in a significant way. For the visual task, it could likely be replaced with a CNN encoder as the Embedding function (Figure 2b), as done in prior work [1], which would reduce computational complexity and likely not hurt performance. I encourage the authors to revisit the choice of why ABA is introduced in the first place.\n\nOne core limitation of the method is its computational complexity. The authors highlight strongly competitive results on tabular data, but this only takes into account the validation error / accuracy. I strongly suspect that the proposed method is by far not competitive with the shown baselines when taking into account computation time and memory requirement. These limitations should be highlighted in the main paper (not in the appendix) by showing a direct comparison in terms of wall clock training/inference time (and memory) to give researchers and practitioners a clear picture about the practicality of the technique. I agree with the authors that alleviating this limitation is a promising avenue for future work.\n\nIn terms of related work, I encourage the authors to use the additional page for the camera-ready version (should the paper be accepted) to include the discussion on graph-based semi-supervised learning using graph neural networks in the main paper (instead of the appendix), as the closest relatives to the proposed model come from this research area. This includes a highly-cited paper [1] which similarly introduces an attention mechanism that  aggregates information across data points for semi-supervised learning and few-shot learning in an architecture that is closely related to the one presented here. This paper is currently not cited/discussed, but I think it should be closely contrasted against the proposed technique.\n\nOther suggestions for improvement:\n* The CIFAR-10 and MNIST results are currently very difficult to interpret in the absence of any baselines. A comparison against a variant of the model that does not use attention between data points (in the main paper) would significantly strengthen this section.\n* The statement that \"they can learn to completely ignore other inputs, essentially collapsing into a standard parametric model\", in the context of perturbed features (Section 4.3) should be verified, e.g. by comparing the attention weights with and without perturbation of the features.\n\n[1] Garcia & Bruna, Few-Shot Learning with Graph Neural Networks (ICLR 2018)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation of the method -- its computational complexity -- is briefly discussed in Section 5. It could be valuable to highlight this limitation more prominently by showing explicit wall clock time and memory usage comparisons against baselines. \n\nThe authors discuss societal impacts at an impressive level of detail in the appendix.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "iUE_ZKFh4le", "reviewer": "Reviewer_iTJu", "summary": "Deep parametric models have demonstrated tremendous success in NLP, computer vision, and many other settings. These models typically take as input an instance and output a prediction / label(s) for that instance. In this paper, the authors describe an approach that stands in contrast, the input to the model is the _entire_ dataset. Predictions are made collectively via attention between (1) data points and (2) attributes. Unlike non-parametric models that solely interpolate between instances, the proposed Non-Parametric Transformers (NPTs) learn attention mechanisms between data points and between attributes. \n\nThe authors describe how the NPT can be trained with a masking approach analogous to masked language-models as well as an approach analogous to 'standard' classification settings. \n\nThe authors provide extensive experiments that not only demonstrate the empirical effectiveness on established benchmarks tasks, but also to better understand the behavior and capabilities of each component of the NPT model. ", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The intersection of non-parametric models and deep parametric models is an area that has garnered much attention (and rightfully so, in my opinion). Works such as Neural Processes, Deep Gaussian Processes, and Deep Kernel Learning have had significant impact. The proposed NTP model advances these previous works in significant, not incremental, ways. It provides a novel architecture, which has capabilities not achieved by previous works. In particular, I believe the paper offers the following merits:\n\n* **Innovative Architecture** - The NPT architecture is novel and interesting and offers new perspectives on the ways to design and parameterize deep transformer-like architectures.\n* **Inspiration for Future Work** - The paper has the potential to lead to future work both in advancing NPT-like methodologies as well as applications of NPT to a wide array of other tasks. \n* **Empirical Analysis** - The authors carefully design experiments to test the performance of the model in different characteristics of the model. These provide insights into how the model behaves in particular circumstances. \n* **Well written** - The manuscript is very clearly written and is a pleasure to read. The figures and illustrations are clear and informative. The experimental hypotheses are clearly defined and experiments are illustrative of the merits of the approach. The description of the model is also clear and well motivated.  \n\nOverall, I think that this is a very strong paper. Here are some ways in which I think it could be improved: \n\n* **Retrieve and Edit** - There are seemingly some similarities between approaches such as retrieve-and-edit models that retrieve relevant instances and transform them to make a prediction for a target instance, e.g. (Hashimoto et al, 2018), inter-alia. It would be interesting to explore the relationship between such approaches and NPT. \n* **Transductive, Case-based, Label Propagation** - Classic approaches such as transductive learning or case-based reasoning I believe are also quite relevant to NPT. Both of these paradigms have also seen a recent revival combining deep learning models and transductive / case-based approaches. It would also be interesting to understand relationships between label propagation models (say on fully connected graphs) and the NPT model. \n* **Structured Prediction** - Many structured prediction tasks can naturally extended their associated models / objectives / energy functions to the dataset level, it might be interest to understand relationships between NPT and structured prediction models. \n* **Differences from Interpolation** - I am a bit unsure that I follow the argument in lines 170-172: \"For example, NPTs could learn to assign test datapoints to clusters of training datapoints, and predict on those points using interpolation of the training targets in their respective cluster. We explore the ability of NPTs to solve such complex reasoning tasks in §4.2.\" . In 4.2, what I am having trouble with is why this is a reasoning task rather than an interpolation task. Isn't an interpolative to put all interpolation weight on the unmasked 'copy' of the masked instance?\n* **Theoretical results** - I don't think this is a requirement of the paper. I think that the empirical analysis is illustrative of the method. The impact of the paper could perhaps be improved with a theoretical analysis of the method.\n\n**Update after author response** Thanks very much to the authors for their response. I believe this is a interesting paper with many notable merits. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe the authors do a good job with this. Perhaps, additional considerations / notes could be added to address how the potential for biases in the model might manifest differently in NPT compared to standard parametric models. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "dNAUsm3s6f8", "reviewer": "Reviewer_bbzv", "summary": "This paper proposes to use self-attention between data points, in order to model the relation between the samples in the dataset. The big claim of the paper is that by doing so, they are going beyond individual input-output pairs in deep learning. While true, this is the main paradigm of entire fields like person re-identification and metric learning, so this claim is massively overstated.\n\nThe paper presents good results in tabular datasets, reaching the best results in 4 out of 10 datasets, which is very good considering that different versions of boosting dominate the field.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The paper reaches very good results in tabular datasets. They reach the best results in 4 out of 10 datasets, better than any other method. When separated into the three main tasks: binary classification, multi-class classification, and regression, they reach the best results in binary classification, joined best results in multi-class classification, and joined second-best results in regression. Overall, it can be considered that the method works at least on par (in my opinion, slightly better) than the two main gradient boosting methods: XGBoost and CatBoost.\n\n- Understanding the method is quite straightforward, the authors do a good job at explaining it.\n\n- The authors do a good job at giving insights after they present their method.\n\nWeaknesses:\n\n1) Massively overstated claim. The first sentence of the paper 'We challenge a common assumption underlying most supervised deep learning: that a model makes a prediction depending only on its parameters and the features of a single input.' gives the impression that this is the first paper that tries to model some sort of relation between the data, instead of input-output relation which is commonly done in deep learning. However, this is a very misleading statement, contextual relations between the data in neural networks have been going on at least since RNNs were invented. More interestingly, there are entire fields in deep learning whose sole purpose is to model the relations between the data. Those fields fall under the umbrella of similarity learning, and are specialized in different directions like metric learning, person re-identification, face recognition etc.\n\nIn all those fields, the idea is that the embedding/results of a data point are not dependent only on itself, but also on the other data points. Typically that was done using some sort of contrastive/triplet loss, but nowadays that is done by using the relations between all the pairs in the dataset. That could be done as some sort of ranking, proxy, or contextual procedure. For example:\n\n[A] Wang et al., Ranked List Loss for Deep Metric Learning, CVPR 2019 (Ranking)\n\n[B] Movshovitz-Attias et al., No Fuss Distance Metric Learning using Proxies, ICCV 2017 (Proxy)\n\n[C] Kim et al., Proxy Anchor Loss for Deep Metric Learning, CVPR 2020 (Proxy)\n\n[D] Elezi et al., The Group Loss for Deep Metric Learning, ECCV 2020 (Contextual)\n\n[E] Bao et al., Masked Graph Attention Network for Person Re-identification, CVPRW2019 (Contextual)\n\n[F] Seidenschwarz et al., Learning Intra-Batch Connections for Deep Metric Learning, ICML 2021 (Contextual)\n\nTake a look especially on papers [E, F]. They are doing almost the same thing as this paper, a rewriting of the GAT paper [G] to solve some other task. [F] specifically mentions that they use Transformers to model the relations between the samples in the dataset, which is the same thing as this paper.\n\n[G] Velickovic et al., Graph Attention Networks, ICLR 2018.\n\nYet, this is not discussed at all in this paper (although [G] has been cited, but not properly discussed). In many ways, the paper is not doing anything more than these other papers, making the claim massively exaggerated. \n\n2) Misleading statement - \n\nline 62-63: We provide the model with the entire dataset – all datapoints – as input.\nline 72: NPTs take as input the entire dataset $X \\in R^{n×d}$.\n\nMy first impression was how is this scalable, and I was expecting a sophisticated way of somehow memorizing the dataset or using some hashing procedure that allows that. To my disappointment, it quickly became clear that the authors, in fact, do not use the entire dataset. Instead, they use minibatches of the dataset (lines 173-179), something that is done in virtually every other deep learning paper (including the ones I mentioned above).\n\n3) Very poor results in CIFAR10/MNIST: I was initially disappointed that when it comes to image datasets, the method is tested only in very simple and long-outdated datasets, like MNIST and CIFAR-10. I would have expected that 'a general framework' that goes 'beyond deep learning' would actually do some more realistic evaluations in more complicated datasets. However, I was even more disappointed to see the results in CIFAR10, where the method reaches only 68.2%. CIFAR10 is a dataset where SOTA is over 99.5% reached by a method that uses Transformers [H]. Even compared to only non-parametric methods (which the method claims to be, although it is a mix between parametric and non-parametric), the results in other works are considerably better (75.89% [I]). Similarly, the method reaches only 98.3% accuracy in MNIST, way below other papers that reach 99.8% (including more than a decade old work of [J]) or the non-parametric method of [I] that reaches 99.44%.\n\n[H] Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021\n\n[I] Blomqvist et al., Deep Convolutional Gaussian Processes, ECML-PKDD 2019\n\n[J] Cireşan et al., Multi-column Deep Neural Networks for Image Classification, CVPR 2012\n\nVerdict: If the paper would have been presented for what it is: a paper that combines ideas from other fields to progress the task of classification/regression in tabular datasets, this paper would have been an interesting one, and probably deserving of getting published at NeurIPS. However, the claims of the paper are massively exaggerated, the related work does not mention other papers that roughly do the same thing (in more complicated tasks), making the abstract, introduction, and conclusions completely misleading. Adding to that, the weak results in image datasets, I do not see the paper being fit for publishing at the NeurIPS conference. My low score is quite heavily weighted towards the misleading parts of the paper and it promising far more than it shows.\n\n**Update:** Please see the response to rebuttal for the updated version of the review, and why I changed my original score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed. The paper is a general framework, so the potential negative societal impact does not apply.", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "7euIu5kyMV-", "reviewer": "Reviewer_miRP", "summary": "This paper has the following contributions:\n* Propose Non-Parametric Transformers (NPT), a trained transformer applying attention across training data points to make predictions.\n* Demonstrate that attention across datapoints is crucial for NPTs.\n* Display good results of transformers on tabular datasets.\n* Apply a self-supervised objective to tabular data by masking some features.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find this paper very interesting and well written. The proposed model is novel and performs well on tabular data. The authors conduct extensive experiments to test the hypothesis that the network leverage the access to training datapoints and does not only use the model's weights. \n\nIt is not easy to place the contribution in the litterature and the authors make a significant effort of exposing related work. I would also relate to [1] where the model attend to a (learned) memory which could be seen in your case as the training dataset itself. \n\n**Precision on the NPT architecture.**\n\"we alternatingly apply attention between datapoints, and attention between attributes of individual datapoints\"\nIt would be helpful to explicitely state if you are using or not MLPs between these layers as this is standard in transformers.\n\nThe attention mechanism between attributes is interesting and could be used to cope with variable number of features or repeated features. Unfortunately, these benefits are annealed by the flattening and use of the $d \\cdot e$ size vector for ABD. An ablation of ABA (Attention Between Attributes) would be very informative. I suspect that a simple MLP on the features/channels would work as well.\n\n**Evaluation on Image Data.** The authors are showing \"early results\" on images that are far from significant. Namely, 68.2% accuracy on CIFAR and 98.3% accuracy on MNIST only shows the reader that NPT can be run on images but this is not a contribution. They should at least include a direct comparison to [2] which achieves 87.1% on CIFAR10 with a small transformer and no pretraining.\n\nI found Appendix C.3 on embedding of tabular features very informative and maybe it could be extended in the main text.\n\nThe feature masking objective seems interesting and I am curious if it is crucial for the success of NPT. Could you run an ablation study with $\\lambda = 0$?\n\n\n[1] Augmenting Self-attention with Persistent Memory\nSainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, Armand Joulin\nhttps://arxiv.org/pdf/1907.01470.pdf\n\n[2] On the Relationship between Self-Attention and Convolutional Layers\nJean-Baptiste Cordonnier, Andreas Loukas, Martin Jaggi\nICLR 2020\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores alternative modeling paradigms for supervised deep learning. In this paper, the authors describe an approach wherein input to the model is the entire dataset along with the query instance. Predictions are made collectively via attention between (1) data points and (2) attributes. This model which attends over entire dataset for predication is named Non-Parametric Transformers (NPTs). The authors describe how the NPT can be applied in various settings, such as classification. Through extensive experiments and ablation studies, the effectiveness of NPT is demonstrated on established benchmarks tasks. Moreover, I (and all the reviewers) found the paper well-written and very clear in its exposition and idea presentation. 

We thank the reviewers and authors for engaging in an active discussion, which resulted in clearing a lot of the concerns (e.g. speed/flops) and a lot of constructive feedback were provided to improve the paper. The authors provided extensive new empirical results/ablation studies as part of the discussion, please include them in the final version of the paper as they add great value and understanding to the model as a whole. Overall reviewers reached to the consensus that the paper has many merits and should be accepted.
Decision: Accept (Poster)