Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Deep parametric models have demonstrated tremendous success in NLP, computer vision, and many other settings. These models typically take as input an instance and output a prediction / label(s) for that instance. In this paper, the authors describe an approach that stands in contrast, the input to the model is the _entire_ dataset. Predictions are made collectively via attention between (1) data points and (2) attributes. Unlike non-parametric models that solely interpolate between instances, the proposed Non-Parametric Transformers (NPTs) learn attention mechanisms between data points and between attributes. 

The authors describe how the NPT can be trained with a masking approach analogous to masked language-models as well as an approach analogous to 'standard' classification settings. 

The authors provide extensive experiments that not only demonstrate the empirical effectiveness on established benchmarks tasks, but also to better understand the behavior and capabilities of each component of the NPT model. 
main_review: The intersection of non-parametric models and deep parametric models is an area that has garnered much attention (and rightfully so, in my opinion). Works such as Neural Processes, Deep Gaussian Processes, and Deep Kernel Learning have had significant impact. The proposed NTP model advances these previous works in significant, not incremental, ways. It provides a novel architecture, which has capabilities not achieved by previous works. In particular, I believe the paper offers the following merits:

* **Innovative Architecture** - The NPT architecture is novel and interesting and offers new perspectives on the ways to design and parameterize deep transformer-like architectures.
* **Inspiration for Future Work** - The paper has the potential to lead to future work both in advancing NPT-like methodologies as well as applications of NPT to a wide array of other tasks. 
* **Empirical Analysis** - The authors carefully design experiments to test the performance of the model in different characteristics of the model. These provide insights into how the model behaves in particular circumstances. 
* **Well written** - The manuscript is very clearly written and is a pleasure to read. The figures and illustrations are clear and informative. The experimental hypotheses are clearly defined and experiments are illustrative of the merits of the approach. The description of the model is also clear and well motivated.  

Overall, I think that this is a very strong paper. Here are some ways in which I think it could be improved: 

* **Retrieve and Edit** - There are seemingly some similarities between approaches such as retrieve-and-edit models that retrieve relevant instances and transform them to make a prediction for a target instance, e.g. (Hashimoto et al, 2018), inter-alia. It would be interesting to explore the relationship between such approaches and NPT. 
* **Transductive, Case-based, Label Propagation** - Classic approaches such as transductive learning or case-based reasoning I believe are also quite relevant to NPT. Both of these paradigms have also seen a recent revival combining deep learning models and transductive / case-based approaches. It would also be interesting to understand relationships between label propagation models (say on fully connected graphs) and the NPT model. 
* **Structured Prediction** - Many structured prediction tasks can naturally extended their associated models / objectives / energy functions to the dataset level, it might be interest to understand relationships between NPT and structured prediction models. 
* **Differences from Interpolation** - I am a bit unsure that I follow the argument in lines 170-172: "For example, NPTs could learn to assign test datapoints to clusters of training datapoints, and predict on those points using interpolation of the training targets in their respective cluster. We explore the ability of NPTs to solve such complex reasoning tasks in §4.2." . In 4.2, what I am having trouble with is why this is a reasoning task rather than an interpolation task. Isn't an interpolative to put all interpolation weight on the unmasked 'copy' of the masked instance?
* **Theoretical results** - I don't think this is a requirement of the paper. I think that the empirical analysis is illustrative of the method. The impact of the paper could perhaps be improved with a theoretical analysis of the method.

**Update after author response** Thanks very much to the authors for their response. I believe this is a interesting paper with many notable merits. 

limitations_and_societal_impact: I believe the authors do a good job with this. Perhaps, additional considerations / notes could be added to address how the potential for biases in the model might manifest differently in NPT compared to standard parametric models.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper introduces a neural network architecture termed Non-Parametric Transformer (NPT) that uses self-attention (i) between data points and (ii) between features within each data point. The core conceptual contribution is to treat supervised deep learning problems as tasks where the model receives the entire dataset (incl. both labeled and unlabeled data points) as input — in contrast to the common approach of providing data samples independently to a deep learning model. The proposed NPT model is evaluated on tabular datasets where the authors report competitive results compared to boosted decision trees (which represent the state of the art). Applied on MNIST and CIFAR10, the NPT model performs reasonably well. The authors further perform a series of experiments to analyze the learned attention patterns and the importance of self-attention between data points for generalization performance.
main_review: This is a high-quality paper that is exceptionally well-written, well-structured and very clear in its exposition. The idea is novel and well-positioned against related work (with some exceptions). I think that this paper is relevant for the NeurIPS community and I expect that it will inspire a range of follow-up works.

The idea of using self-attention to take into account all other data points (including their labels, if available) both during training and inference is intriguing, and given the rising popularity Transformer-style architectures in the machine learning community, I expect that this method will spawn lots of interest despite its shortcomings in its current form, such as its computational complexity and its limited usefulness/applicability in any of the studied tasks in the paper.

The experimental evaluation is carried out with great care and the authors come up with several insightful experiments to study whether the attention between datapoint (ABD) mechanism learns to make use of features/labels of similar data points.

Overall, I think that this paper can be accepted and that it would make a great addition to the conference program. I do, however, want to highlight several limitations of the method and issues with the paper below, which I encourage the authors to take into account in the next version of the paper.

The core contribution of the paper is the attention between datapoint (ABD) mechanism and its application for semi-supervised learning in tabular data. While the other model contribution, namely attention between attributes (ABA) is interesting, it is not properly experimentally evaluated (e.g. against an alternative, simpler architecture component, such as an MLP on the flattened representation, or simply the identity, i.e. no module at all). The reader might wonder why this module is introduced in the first place, and its necessity is unclear. I could imagine that this module could simply be left out for the tabular tasks without affecting performance in a significant way. For the visual task, it could likely be replaced with a CNN encoder as the Embedding function (Figure 2b), as done in prior work [1], which would reduce computational complexity and likely not hurt performance. I encourage the authors to revisit the choice of why ABA is introduced in the first place.

One core limitation of the method is its computational complexity. The authors highlight strongly competitive results on tabular data, but this only takes into account the validation error / accuracy. I strongly suspect that the proposed method is by far not competitive with the shown baselines when taking into account computation time and memory requirement. These limitations should be highlighted in the main paper (not in the appendix) by showing a direct comparison in terms of wall clock training/inference time (and memory) to give researchers and practitioners a clear picture about the practicality of the technique. I agree with the authors that alleviating this limitation is a promising avenue for future work.

In terms of related work, I encourage the authors to use the additional page for the camera-ready version (should the paper be accepted) to include the discussion on graph-based semi-supervised learning using graph neural networks in the main paper (instead of the appendix), as the closest relatives to the proposed model come from this research area. This includes a highly-cited paper [1] which similarly introduces an attention mechanism that  aggregates information across data points for semi-supervised learning and few-shot learning in an architecture that is closely related to the one presented here. This paper is currently not cited/discussed, but I think it should be closely contrasted against the proposed technique.

Other suggestions for improvement:
* The CIFAR-10 and MNIST results are currently very difficult to interpret in the absence of any baselines. A comparison against a variant of the model that does not use attention between data points (in the main paper) would significantly strengthen this section.
* The statement that "they can learn to completely ignore other inputs, essentially collapsing into a standard parametric model", in the context of perturbed features (Section 4.3) should be verified, e.g. by comparing the attention weights with and without perturbation of the features.

[1] Garcia & Bruna, Few-Shot Learning with Graph Neural Networks (ICLR 2018)

limitations_and_societal_impact: The main limitation of the method -- its computational complexity -- is briefly discussed in Section 5. It could be valuable to highlight this limitation more prominently by showing explicit wall clock time and memory usage comparisons against baselines. 

The authors discuss societal impacts at an impressive level of detail in the appendix.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper has the following contributions:
* Propose Non-Parametric Transformers (NPT), a trained transformer applying attention across training data points to make predictions.
* Demonstrate that attention across datapoints is crucial for NPTs.
* Display good results of transformers on tabular datasets.
* Apply a self-supervised objective to tabular data by masking some features.
main_review: I find this paper very interesting and well written. The proposed model is novel and performs well on tabular data. The authors conduct extensive experiments to test the hypothesis that the network leverage the access to training datapoints and does not only use the model's weights. 

It is not easy to place the contribution in the litterature and the authors make a significant effort of exposing related work. I would also relate to [1] where the model attend to a (learned) memory which could be seen in your case as the training dataset itself. 

**Precision on the NPT architecture.**
"we alternatingly apply attention between datapoints, and attention between attributes of individual datapoints"
It would be helpful to explicitely state if you are using or not MLPs between these layers as this is standard in transformers.

The attention mechanism between attributes is interesting and could be used to cope with variable number of features or repeated features. Unfortunately, these benefits are annealed by the flattening and use of the $d \cdot e$ size vector for ABD. An ablation of ABA (Attention Between Attributes) would be very informative. I suspect that a simple MLP on the features/channels would work as well.

**Evaluation on Image Data.** The authors are showing "early results" on images that are far from significant. Namely, 68.2% accuracy on CIFAR and 98.3% accuracy on MNIST only shows the reader that NPT can be run on images but this is not a contribution. They should at least include a direct comparison to [2] which achieves 87.1% on CIFAR10 with a small transformer and no pretraining.

I found Appendix C.3 on embedding of tabular features very informative and maybe it could be extended in the main text.

The feature masking objective seems interesting and I am curious if it is crucial for the success of NPT. Could you run an ablation study with $\lambda = 0$?


[1] Augmenting Self-attention with Persistent Memory
Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, Armand Joulin
https://arxiv.org/pdf/1907.01470.pdf

[2] On the Relationship between Self-Attention and Convolutional Layers
Jean-Baptiste Cordonnier, Andreas Loukas, Martin Jaggi
ICLR 2020

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to use self-attention between data points, in order to model the relation between the samples in the dataset. The big claim of the paper is that by doing so, they are going beyond individual input-output pairs in deep learning. While true, this is the main paradigm of entire fields like person re-identification and metric learning, so this claim is massively overstated.

The paper presents good results in tabular datasets, reaching the best results in 4 out of 10 datasets, which is very good considering that different versions of boosting dominate the field.
main_review: Strengths:

- The paper reaches very good results in tabular datasets. They reach the best results in 4 out of 10 datasets, better than any other method. When separated into the three main tasks: binary classification, multi-class classification, and regression, they reach the best results in binary classification, joined best results in multi-class classification, and joined second-best results in regression. Overall, it can be considered that the method works at least on par (in my opinion, slightly better) than the two main gradient boosting methods: XGBoost and CatBoost.

- Understanding the method is quite straightforward, the authors do a good job at explaining it.

- The authors do a good job at giving insights after they present their method.

Weaknesses:

1) Massively overstated claim. The first sentence of the paper 'We challenge a common assumption underlying most supervised deep learning: that a model makes a prediction depending only on its parameters and the features of a single input.' gives the impression that this is the first paper that tries to model some sort of relation between the data, instead of input-output relation which is commonly done in deep learning. However, this is a very misleading statement, contextual relations between the data in neural networks have been going on at least since RNNs were invented. More interestingly, there are entire fields in deep learning whose sole purpose is to model the relations between the data. Those fields fall under the umbrella of similarity learning, and are specialized in different directions like metric learning, person re-identification, face recognition etc.

In all those fields, the idea is that the embedding/results of a data point are not dependent only on itself, but also on the other data points. Typically that was done using some sort of contrastive/triplet loss, but nowadays that is done by using the relations between all the pairs in the dataset. That could be done as some sort of ranking, proxy, or contextual procedure. For example:

[A] Wang et al., Ranked List Loss for Deep Metric Learning, CVPR 2019 (Ranking)

[B] Movshovitz-Attias et al., No Fuss Distance Metric Learning using Proxies, ICCV 2017 (Proxy)

[C] Kim et al., Proxy Anchor Loss for Deep Metric Learning, CVPR 2020 (Proxy)

[D] Elezi et al., The Group Loss for Deep Metric Learning, ECCV 2020 (Contextual)

[E] Bao et al., Masked Graph Attention Network for Person Re-identification, CVPRW2019 (Contextual)

[F] Seidenschwarz et al., Learning Intra-Batch Connections for Deep Metric Learning, ICML 2021 (Contextual)

Take a look especially on papers [E, F]. They are doing almost the same thing as this paper, a rewriting of the GAT paper [G] to solve some other task. [F] specifically mentions that they use Transformers to model the relations between the samples in the dataset, which is the same thing as this paper.

[G] Velickovic et al., Graph Attention Networks, ICLR 2018.

Yet, this is not discussed at all in this paper (although [G] has been cited, but not properly discussed). In many ways, the paper is not doing anything more than these other papers, making the claim massively exaggerated. 

2) Misleading statement - 

line 62-63: We provide the model with the entire dataset – all datapoints – as input.
line 72: NPTs take as input the entire dataset $X \in R^{n×d}$.

My first impression was how is this scalable, and I was expecting a sophisticated way of somehow memorizing the dataset or using some hashing procedure that allows that. To my disappointment, it quickly became clear that the authors, in fact, do not use the entire dataset. Instead, they use minibatches of the dataset (lines 173-179), something that is done in virtually every other deep learning paper (including the ones I mentioned above).

3) Very poor results in CIFAR10/MNIST: I was initially disappointed that when it comes to image datasets, the method is tested only in very simple and long-outdated datasets, like MNIST and CIFAR-10. I would have expected that 'a general framework' that goes 'beyond deep learning' would actually do some more realistic evaluations in more complicated datasets. However, I was even more disappointed to see the results in CIFAR10, where the method reaches only 68.2%. CIFAR10 is a dataset where SOTA is over 99.5% reached by a method that uses Transformers [H]. Even compared to only non-parametric methods (which the method claims to be, although it is a mix between parametric and non-parametric), the results in other works are considerably better (75.89% [I]). Similarly, the method reaches only 98.3% accuracy in MNIST, way below other papers that reach 99.8% (including more than a decade old work of [J]) or the non-parametric method of [I] that reaches 99.44%.

[H] Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021

[I] Blomqvist et al., Deep Convolutional Gaussian Processes, ECML-PKDD 2019

[J] Cireşan et al., Multi-column Deep Neural Networks for Image Classification, CVPR 2012

Verdict: If the paper would have been presented for what it is: a paper that combines ideas from other fields to progress the task of classification/regression in tabular datasets, this paper would have been an interesting one, and probably deserving of getting published at NeurIPS. However, the claims of the paper are massively exaggerated, the related work does not mention other papers that roughly do the same thing (in more complicated tasks), making the abstract, introduction, and conclusions completely misleading. Adding to that, the weak results in image datasets, I do not see the paper being fit for publishing at the NeurIPS conference. My low score is quite heavily weighted towards the misleading parts of the paper and it promising far more than it shows.

**Update:** Please see the response to rebuttal for the updated version of the review, and why I changed my original score.

limitations_and_societal_impact: Limitations are discussed. The paper is a general framework, so the potential negative societal impact does not apply.
review_title: null

Meta-review: This paper explores alternative modeling paradigms for supervised deep learning. In this paper, the authors describe an approach wherein input to the model is the entire dataset along with the query instance. Predictions are made collectively via attention between (1) data points and (2) attributes. This model which attends over entire dataset for predication is named Non-Parametric Transformers (NPTs). The authors describe how the NPT can be applied in various settings, such as classification. Through extensive experiments and ablation studies, the effectiveness of NPT is demonstrated on established benchmarks tasks. Moreover, I (and all the reviewers) found the paper well-written and very clear in its exposition and idea presentation. 

We thank the reviewers and authors for engaging in an active discussion, which resulted in clearing a lot of the concerns (e.g. speed/flops) and a lot of constructive feedback were provided to improve the paper. The authors provided extensive new empirical results/ablation studies as part of the discussion, please include them in the final version of the paper as they add great value and understanding to the model as a whole. Overall reviewers reached to the consensus that the paper has many merits and should be accepted.
Decision: Accept (Poster)