Title: Fast Training of Large Kernel Models with Delayed Projections
Venue: NeurIPS 2025
Accepted: yes
Keywords: Kernel machines

large-scale kernel machines

Preconditioned-SGD

Nyström approximation
Abstract: Classical kernel machines have historically faced significant challenges in scaling to large datasets and model sizes—a key ingredient that has driven the success of neural networks. In this paper, we present a new methodology for building kernel machines that can scale efficiently with both data size and model size. Our algorithm introduces delayed projections to Preconditioned Stochastic Gradient Descent (PSGD) allowing the training of much larger models than was previously feasible. We validate our algorithm, \EP4, across multiple datasets, demonstrating drastic training speedups without compromising the performance. Our implementation is publicly available at: https://github.com/EigenPro/EigenPro
.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose EigenPro4, the algorithm for fast training of kernel machines using Preconditioned Stochastic Gradient Descent. This work builds upon the previous versions of the algorithms: EigenPro, EigenPro2, and EigenPro3. The key trick is to use a subset of training points to represent the function, rather than using all trainingp points. The key difficulty is that the computed gradient in each iteration lies in the span of the training points, not in the span of the chosen Nystrom centers. In order to maintain the iterate remains in the span of the Nystrom centers (low rank), a projection method which is the most expensive step ($O(p^2)$ per iterate) was proposed in EigenPro3. This projection step was improved to $O(p)$ in EigenPro4 by delaying the projections for $T$ iterations. This means in-between the two successive projection steps, the prediction function has additional temporary centers; so for $T$ delayed projections, the maintained centers would be expanded by at most $T m$ additional centers. The authors compare EigenPro4 against EigenPro3 and Falkon on real-world datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The manuscript is clearly written and the authors have actually made conscientious efforts to limit unnecessarily complex notations to aid in understanding the algorithm. Experimental evaluations are thorough and enlightening, although they do not cover every variant of fast kernel machines.\n\nWeaknesses: The improvement is rather trivial and relies on delaying the projection step to improve efficiency of the algorithm. But perhaps this triviality makes the manuscript stand out in times when many obscure papers are abundant.\n\nQUESTIONS:\n1. I could not find what parameters were used for EigenPro4 in the experiment. Could the authors clarify?\n2. In general, the results for each method used should be parameterized (e.g. Table 1). In general, it is hard to find these parameters within the writeup.\n3. Would it be possible to incorporate a nearest neighbor heuristic within Algorithm 1 (Line 8) to further maintain a more compress set of centers?\n4. Other than the Laplace kernel (again, I was not able to find which kernel was being used for the experiment in the captions), what other kernels have the authors tried?\n\nLIMITATIONS:\nThere are no significant limitations in the authors' work. However, while the experimental results for the most part are clearly presented, the authors are encouraged to tabulate the parameters used for each method for clarity.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel algorithm EigenPro 4 for training large kernel models.\nThe algorithm is an upgrade an existing EigenPro 3 version which uses preconditioned stochastic gradient descent with certain projections in order to decouple the model size from the training set size. This projection requires quadratic time in the model size and so far had to be computed after each stochastic step which yields the bottleneck for the running time.\nThe authors overcome this bottleneck by delaying the projections for $T$ iterations, where $T$ is a predefined hyperparameter (it is claimed that the optimal value for $T$ is the ratio of model size and batch size).\n\nThey validate their approach empirically with experiments on three datasets which show a clear speedup of the EigenPro 4 version compared to EigenPro3 and another state-of-the art method.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: The paper presents a natural approach to speed up the computational performance of the state-of-the art of kernel based learning. The performance seems to be considerably improved at least on the investigated datasets. Also the convergence rate was faster.\n\nWeakness: The result is very incremental. The idea to delay projections seems rather obvious and of little originality. The increase in performance looks solid, but I am not sure about the significance for practical purposes (since I am not an expert in practical applications).\nAlso, there is no theoretical analysis provided that guarantees convergence.\nFinally, the presentation could be improved. Many things are not defined or explained and there are several language errors. (See comments below.)\n\nComments:\n- l 28: \"to improve condition number\" -> improve the condition number\n- l 51: \"on CIFARM data set\" -> on the CIFARM data set\n- l 63: z_i not mathbold. Also explain the idea behind these centers of a general kernel model.\n- l 65: Define what a RKHS is.\n- Fig 1: Explain notation p.\n- l 80:  Explain the tensor product sign.\n- l 95: \"condition number is ill conditioned\" -> This is a confusing formulation.\n- l 102: \"unit-norm e_i\" -> Do you mean standard unit vector?\n- l 104: \"let ... as the\" -> let ... be the\n- l 134: \"models of this form\" -> which form are you referring to?\n- l 136: The variable m is not defined yet. (Also you use m later for mini batches (X_m,y_m), which could be confusing)\n- l 150: \"add to Z_tmp with temporary centers\" -> add to Z_tmp the temporary...\n- Fig 3: What is n?\n\nQUESTIONS:\n1) You mention that you prove convergence of Alg 1 for $T=\\infty$ in the appendix. This would mean that it never projects, which does not really make sense. In fact, you prove it for $T=1$?\n\n2) How did you choose $T$ and all other hyperparameters in your experiments?\n\n3) Do you have an explanation why also the convergence rate is faster?\n\n4) Can your algorithm compete with other models?\n\nLIMITATIONS:\nThe paper does not really discuss any limitations of the approach.\nFor example, how does the algorithm perform in comparison to other (non-kernel) machine learning methods.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work (EigenPro 4) proposes Delayed Projection, instead of projection after each gradient descent step, to address high time cost per batch of EigenPro 3. Convergence and time complexity analysis are given. Experiments show the great reduction in per-epoch training time and good scalability w.r.t. model size.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe motivation of delayed projection is clear and easy to understand.\n\n2.\tThe analysis of convergence and time complexity is thorough and sound.\n\n3.\tThe experiment results demonstrate that the proposed method is practical on very large datasets . Compared with EigenPro 3, the time reduction does not cause severe test accuracy drop.\n\nWeaknesses:\n\n1.\tIn terms of effectiveness, EigenPro 4 performs inconsistently across different model sizes: it is strange to observe that although EigenPro 4 is more efficient than EigenPro 3, the accuracy of EigendPro 4 is lower (higher) than EigenPro 3 when the model size is small (large). Is there any particular reason behind this? \n\n2.\tAlthough EigenPro 4 is faster than EigenPro 3 and converges faster. The results in Figure 5 indicate the convergence process of EigenPro 4 is more unstable than EigenPro 3 due to delayed projection, especially in Webvision dataset. Is this because EigenPro 4 accuracy improves between projections and drops after each projection step as shown in Figure 5? \n\n3.\tAs claimed in line 727, the EigenPro 4 accuracy drops from projections become progressively smaller. However, in the Webvision and Librispeech results in Figure 5, the multiple accuracy drops remain relatively large when EigenPro 4 converges. Is there any specific reason behind this occurrence that warrants further exploration on the stability of EigenPro 4? Adding an analysis on this issue would be nice.\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrevious kernel-based methods have difficulties with scaling to larger model sizes due to the computational cost of projecting gradients into the kernel feature space after every iteration. The authors propose EigenPro 4, which improves upon EigenPro 3, by delaying this projection by T iterations, which reduces the per iteration cost while maintaining performance. This approach also allows the model to be approximately linear on average per epoch in terms of per iteration computations while having linear memory scaling.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper presents a practical contribution to kernel methods which would help allow them to be used in more large scaled environments due to the linear amortized per-iteration cost. They also show impressive performance improvements over EigenPro 3 and Falkon in terms of running time. However, the paper itself, especially the beginning sections, are not the most organized nor standalone. There is little contextualization in the beginning, and no clear reason as to why one would want to use kernel methods even with linear amortized per iteration cost and linear memory scaling. Although, the contributions could be significant, the paper does a poor job phrasing its importance with respect to other non-kernel methods. There are also some things missing that would make for a comphrensive evaluation, such as using more kernels in the addition to the laplace kernel or abalations on the value of $T$. Although, $T$ is proportional to $p/m$, no exact value used is provided for the experiments and testing different values for $T$ would be informative.\n\nQUESTIONS:\nThe motivation needs improvement a lot, such as why to choose a kernel-based method over a non-kernel based method (e.g. deep learning models)?\n \nIn equation 15, the otherwise branch calculates the stochastic gradient using $f_t$, but should it not be on $f_{t-1}$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new method for solving kernel regression via gradient\ndescent. Their method, EigenPro 4, is based heavily upon EigenPro 3. The paper\nidentifies the 'projection step' as a major bottleneck in EigenPro 3, and solve\nthis issue by amortization: instead of projecting every step, they accumulate 'temporary centres'\nand project them every T steps. This modification leads to massive speedups vs.\nEigenPro 3, and (perhaps surprisingly) also improves performance. The speedups\nare especially significant, as this unlocks the ability to use kernel regression\nfor larger datasets than previously practicable.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Authors give a clear and concise exposition of kernel regression and of existing algorithms.\n- Experimental results very clearly show that EP4 is significantly faster and\n  more practical than kernel regressor alternatives.\n\nOverall EP4 is obviously very helpful for any practitioners using kernel regression / EP3.\n\nWeaknesses:\n\n- Experiments/Datasets are a bit artificial. For example, the kernel method uses\n  features extracted from a ResNet/MobileNetV2.\n- It's unclear how the proposed methods performs vs. a NN. Even if the NN is\n  more performant, this still seems relevant to include, considering that the\n  underlying features of all experiments are extracted from a NN.\n- Some experimental details are missing in the main text; for example, basic information like what value\n  of T is used?\n- The paper's results suggest that EP4 is strictly better (in terms of accuracy/loss) than EP3, but more\n  results using further kernels/datasets (only 1 kernel and 4 dataset were used)\n  would be helpful to confirm this.\n- The paper does not explain why EigenPro 4 is better than EigenPro 3, which is\n  an interesting quirk. I would like to see a discussion on why accumulating the temporary centres appears to help.\n\n\nMinor issues:\n\n- Algorithm 1 describes computational details of EigenPro 4. It would be nice to\n  have a 'diff' with EigenPro 3 to clarify the computational differences.\n- In Eq. (1) L is defined differently to L in line 78. Make these consistent.\n- Line 81, 'Operator K' -> 'The Operator K'\n- The meaning of * in line 193 (and elsewhere). Please include this,\n  possibly as a footnote, in the main text.\n\nQUESTIONS:\n- Why do we get such a noticable accuracy/performance boost for EP4 over EP3 in\n  Figure 3, when the only difference between the two is the amortization of the\n  projection? Can you comment on whether the amortization might improve\n  properties of the optimization?\n- Figure 5 shows that performance degrades after the projection step. Are you\n  able to explain this behaviour?\n- Can you provide any preliminary experiments with other kernels (i.e. not just\n  the Laplace kernel)? You mentioned that Laplace has strong empirical\n  performance, but compared to what baseline?\n- Is it possible to perform 'deep kernel' experiments? I.e. to stack several\n  kernel regressors together, and train them jointly with EP4?\n\nLIMITATIONS:\nThe authors discuss theoretical limitations of their algorithm in Section 3.6. However, I would like to see a discussion of practical elements too, such as where EP4 is limited versus other methods (e.g. NNs). For example, as far as I know, EP4 is relying on good underlying representations as input to be useful in practice.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-06-26"}