Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors consider the large-scale implementation of 'neural kernels', kernel
methods that are derived from the architecture of a specific neural network and
the correspondence between randomly-initialized neural networks and gaussian
processes. In contrast to existing large-scale kernel methods for use with
kernels like the RBF kernel, the fact that these neural kernels are
computationally-expensive to compute necessitates additional parallelization.
The authors review methods for large-scale kernel regression, implement their
own method in the context of neural kernels, and show experiments on various
large datasets (CIFAR-5m, Tiny ImageNet) where these kernels achieve
performance beyond past kernel methods. They also demonstrate their approach on
various basic science tasks, where it outperforms existing kernel methods.


strength_and_weaknesses: ## Strengths

- The engineering contribution directly enables new evaluation of neural
  kernels at novel scales, enabling their use on new datasets where they set
  new record for performance by kernel methods.
- The paper is well-written and does a good job of organizing various
  approaches to large-scale kernel regression in section 1.2 and section 2. One
  comes away with an excellent understanding of the landscape of approaches to
  this problem, which should be useful for future work.
- The authors do a good job of showing how the engineering advancements open
  new opportunities for kernel methods. They demonstrate that neural kernels
  achieve excellent performance on molecular prediction/classification tasks
  compared to existing methods, and show that data augmentation can give good
  improvements in performance for kernel methods, given the ability to work
  with larger kernels.

## Weaknesses

- The precise methodological contributions seemed somewhat unclear to me -- is
  the main technical contribution to identify the most relevant existing
  large-scale kernel regression approaches for working with neural kernels (as
  outlined in section 2), and then to develop and implement a new parallelization
  scheme on top of these methods to facilitate computing neural kernels? The
  latter feels like a somewhat limited (but nonetheless significant) technical
  contribution, given that the paper's description of the parallelization scheme
  seems to be mostly restricted to showing Figure 1.
- It might be better to have a demonstration of the method on low-resolution
  datasets other than CIFAR-5m, since this dataset is essentially synthetic. I
  think the authors' experiments on Tiny ImageNet are more than appropriate in
  this connection, I just point this out because the description of CIFAR-5m at
  the bottom of page 5 as "...ensuring that the additional data are sampled
  i.i.d." is not correct and somewhat misleading (since the dataset is
  generated from a trained generative model).
- I think the section about neural scaling laws could be improved
  by a more precise discussion of the issues -- once one is working with linear
  models (whether or not the kernel is "neural"), one can make much more
  precise statements about the asymptotic scaling of the test error as a
  function of the number of samples given various distributional assumptions,
  and it does not seem to be appropriate to me to treat this as a purely
  empirical endeavor.
  These types of questions have been studied in the literature on nonparametric
  regression in RKHSes; under simple distributional models for the data (e.g.,
  the target function is sufficiently smooth and the data come from a
  distribution on a $d$-dimensional manifold), the RKHS associated to a given
  kernel can be characterized in terms of differentiability properties of the
  kernel (which are, in turn, inherited from the architectural choices and the
  random initialization scheme), and the corresponding scaling behavior (i.e.,
  the exponent) of the test error becomes a function of this RKHS, the
  dimension $d$, and the target function's smoothness.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written, with clear presentation of experimental results and
ample references to prior work. The work enables new applications of neural
kernels at scale (Tiny ImageNet) and in various applications (the performance
gains over traditional kernels give justification for the authors' work in
these contexts). 

It would be preferable for reproducibility to have code released to reproduce
experiments, rather than just the architecture specifications used to generate
the kernels (appendix A).


summary_of_the_review: 
The engineering contribution and its consequences are significant, and the
authors verify these well through various experiments. One hopes the code will
be released.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper explores further scaling up preconditioned conjugate gradient (PC-CG) based GP regression by parallelizing the computation of the Gram matrix as a one-off and storing it to disk in "sub-blocks" to be loaded in the RAM in a multi-threaded fashion as/when needed. 

This contrasts with standard PC-CG where the Gram matrix is never computed entirely but rather partially recomputed on the fly as needed. 

The standard PC-CG approach works reasonably well when the kernel is easy to compute (e.g. the square-exponential kernel), but breaks down when the kernel is more complex. 

The authors demonstrate the effectiveness of the proposed approach using neural kernels, especially compared to alternatives consisting of approximating the linear system (as opposed to solving it within an error CG-style), and propose additional empirical studies such as how performance of GP regression under neural kernels scale with data sizes.

strength_and_weaknesses: - **Strength**: Scalability is one of the major hurdles in bridging the gap between  GP methods and deep learning. This paper's attempt to doing so is laudable, and experimental results are encouraging.

- **Weaknesses**: Experimental results seem to be missing error bars/intervals, hardware specs, and running durations. Overall the contribution seems too light to be worthy of publication.



clarity,_quality,_novelty_and_reproducibility: - **Clarity**: The paper reads well.
- **Quality**: Experimental results could be strengthened by adding error bars and running durations.
- **Novelty**: The main idea here, namely to parallelize the computation of the Gram matrix and store it to disk, isn't particularly novel.
- **Reproducibility**: While the authors provided Python code, specifications of hardware used would have eased reproducibility.

summary_of_the_review: Overall the main contribution is too simple to be worthy of a paper and, in any case, empirical evaluation should be strengthened.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper empirically investigates the performance of million-scale kernel regression with infinite-width neural networks using the preconditioned conjugate gradient method from the Gaussian process literature (Wang, 2019). The method is applied to several domains including testing scaling laws on CIFAR-5m, enabling data augmentation on CIFAR-10 by a factor of 20, and sequence/molecule classification tasks using convolutional/graph convolutional architectures.
strength_and_weaknesses: ## Strengths

* The paper makes a nice attempt to scaling infinite-width neural-network kernels. This is an important problem faced by the area and a solution would be of interest to a wide audience. 

* The empirical study is comprehensive and covers several data modalities including images, sequences and graphs, all showing competitive performance against strong baselines.

## Weaknesses

* The results are good, but not surprising. Given the existing work of Wang et al. (2019) that already applies GPs to 1 million data points, the scaling in this work was not that far from reach. In fact, the paper is using the same preconditioning as Wang et al. (2019) and from the text I did not see any additional innovations needed to achieve the 5 million scale. Achieving 91% on CIFAR 10 with data augmentation that increases the training set by a factor of 20 does not seem surprising, either. 

* One big question i had in mind while reading this paper is---what is the central message? The focus in each section seems completely orthogonal to each other and the whole piece of work feels more like an experiment report. Simply showing that infinite-neural-network kernels achieves predictable good (but not state-of-the-art) results does not qualify a scientific contribution. 

* Some details are unclear about the 5 million experiment. What are the tolerance level needed to solve this linear system? One big concern I had about Wang et al. (2019) is that they used a very large tolerance (1 if I remember it correctly) during training and did not bother to get the GP hyperparameters right (partly the reason why the training is super fast/scalable). During test time, to ensure good performance, they had to switch to a much smaller tolerance level. Do you have to do the same? What is total training + test time when compared to a standard sparse method like Nystrom? (Nystrom-like ideas  is not totally infeasible for these kernels, see Deng, Z., Shi, J., & Zhu, J. (2022). NeuralEF: Deconstructing Kernels by Deep Neural Networks. arXiv preprint arXiv:2205.00165 
where they show a nice recovery of NTK using sparse eigenfunctions. 

clarity,_quality,_novelty_and_reproducibility: This paper is very clear in what approach it takes and what the result is. However, it is unclear what the central message is. The originality is limited given the prior work on scaling GPs (the authors might want to expand the discussion on distributed computation to show if there is originality there). 
summary_of_the_review: Given the lack of central message and limited amount of contribution compared to prior literature, I would recommend rejection.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work enables kernel regression with the so-called neural tangent kernel to work with significantly larger datasets compared to before (order of million datapoints) by building on top of the framework of [1] for Gaussian processes. The key difference to [1] stems from the difference of computational costs to obtain the kernel matrix; while for GPs, simpler kernels such as the RBF are often used, calculating the Gram matrix of the NTK associated with a convolutional neural network marks the bottleneck, instead of the linear system that needs to be solved. The authors distribute the computation of the kernel over many devices and store it on disk. To perform the matrix-vector product needed in conjugate gradient descent, the authors use a smart way to read chunks of the kernel efficiently, avoiding thus the need to recompute the kernel at every step. The more powerful method enables experiments that were not possible before, such as large scaling law experiments, experiments on TinyImageNet and heavy data augmentation leading to SOTA kernel performance on CIFAR10. Finally, the authors also show that these kernels can reach very strong performance on smaller datasets including molecular tasks.
strength_and_weaknesses: **Strengths**:
1. The paper is very well-written and very easy to follow. I really enjoyed reading this paper.
2. The resulting methodology in my opinion enables a lot of interesting experiments (provided enough compute) that were simply infeasible before. The obtained results provide a better and more fair comparison between finite and infinite networks, showing for instance that when standard training components such as data augmentation are included for kernels, very competitive accuracies can be achieved with the kernel method as well. It also marks a first step towards evaluating such kernels on ImageNet, which further facilitates comparisons.
3. The experiments regarding scaling laws are very interesting and important as they provide a more fine-grained look into performance. To my knowledge, such experiments are missing in the literature. It would have been even more interesting to compare the kernel scaling laws with the corresponding finite network.

**Weaknesses**
1. The methodology largely relies on previous work (i.e. [1]) with the only difference that the kernel is stored on disk instead of computed on the fly. While the smart reading strategy of the chunks is certainly not trivial, the technical novelty of the approach is somewhat limited.
2. While the protein/molecule experiments are very interesting in their own right, I don’t see how they fit into this paper. Most datasets are only moderately large and the involved methodology developed in this paper is thus not needed to perform accurate inference. 

[1] Ke Wang, Geoff Pleiss, Jacob Gardner, Stephen Tyree, Kilian Q Weinberger, and Andrew Gordon Wilson. Exact gaussian processes on a million data points. In Advances in Neural Information Processing Systems, 2019. 
clarity,_quality,_novelty_and_reproducibility: The paper is very well written and the challenges to move from the Gaussian process setting to the neural tangent kernel are clearly described. The experiments are well-motivated and very relevant to the community. What remained unclear to me is how many GPUs/CPUs, RAM etc are needed for certain types of experiments. How many GPUs does one need to train on CIFAR10 with augmentations for instance? Including these numbers would be helpful for the reader who plans to use the framework for his own experiments. 
As mentioned above, this work is largely built on top of the work [1], which reduces the originality of the framework. The obtained extension still remains non-trivial however.

[1] Ke Wang, Geoff Pleiss, Jacob Gardner, Stephen Tyree, Kilian Q Weinberger, and Andrew Gordon Wilson. Exact gaussian processes on a million data points. In Advances in Neural Information Processing Systems, 2019. 
summary_of_the_review: In summary, I would like to see this work accepted at ICLR. The framework developed in this work enables important experiments with NTK that were infeasible before. Scaling laws, usage of data augmentation and the possibility to scale to larger datasets such as TinyImageNet facilitate comparing finite and infinite width networks, which still remains an open problem.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper combines a series of techniques to improve kernel methods on large-scale datasets and achieves impressive results.
The major concern, shared by all reviewers, is that this paper does not have enough technical details to reproduce the experimental results. The AC agrees and thus recommends rejection. 
justification_for_why_not_higher_score: The major concern, shared by all reviewers, is that this paper does not have enough technical details to reproduce the experimental results. The AC agrees and thus recommends rejection.
Decision: Reject