Title: Kernel Learning with Adversarial Features: Numerical Efficiency and Adaptive Regularization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Kernel learning; adversarial training; generalization bounds
Abstract: Adversarial training has emerged as a key technique to enhance model robustness against adversarial input perturbations. Many of the existing methods rely on computationally expensive min-max problems that limit their application in practice. We propose a novel formulation of adversarial training in reproducing kernel Hilbert spaces, shifting from input to feature-space perturbations.  This reformulation enables the exact solution of inner maximization and efficient optimization. It also provides a regularized estimator that naturally adapts to the noise level and the smoothness of the underlying function. We establish conditions under which the feature-perturbed formulation is a relaxation of the original problem and propose an efficient optimization algorithm based on iterative kernel ridge regression. We provide generalization bounds that help to understand the properties of the method. We also extend the formulation to multiple kernel learning. Empirical evaluation shows good performance in both clean and adversarial settings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new method for adversarial training in the regenerated kernel Hilbert space. By transferring perturbations from the input space to the feature space, convex optimization and adaptive regularization are achieved. Theoretical analysis and experiments verified the validity of the method.\n\nSTRENGTHS AND WEAKNESSES:\nThe overall framework is theoretically innovative, but the experimental design and writing expression need to be further improved.\n\n1.The upper bound relationship of the characteristic perturbation objective function with respect to the input perturbation target was derived, and the generalization error limit was derived.\n\n2.Adversarial training migrates from the input space to the feature space, solving the computational bottleneck of the traditional min-max problem.\n\n3.Extend the framework to multiple kernel learning, and the mathematical proof is rather rigorous.\n\nQUESTIONS:\n1.The comparison of related jobs (such as TRADES, etc.) is not comprehensive enough, and the Adversarial training methods in recent years have not been fully discussed in Adversarial training in deep learning, which weakening the positioning of the methods in the field of deep learning\n\n2.In the introduction, a comparison of non-RKHS methods (such as adversarial training in deep learning) is added to highlight the theoretical uniqueness of the method proposed in this paper.\n\n3. The inheritance and development relationship with the work of Ribeiro et al. (2023) can be clearly stated to avoid repetition.\n\n4. If only PGD attacks are tested, why not verify the robustness against other attacks (such as FGSM, C&W)?\n\n5. How sensitive δ and λ are to the results, and the hyperparameter selection criteria (such as the specific implementation effect of cross-validation) are not specified. The influence of the ε value in η-trick on the convergence rate and stability has not been fully discussed.\n\n6. The alternating use of \"adversarial perturbation\" and \"feature perturbation\" can easily lead to misunderstandings. It is recommended to review the entire text and improve the manuscript.\n\n7. “As shown in Figure 1...” However, the content of Figure 1 was not explicitly mentioned in the paper.\n\nLIMITATIONS:\nThe paper is valuable in terms of methodological innovation and theoretical analysis, but it does not fully discuss the technical limitations and neglects the analysis of social impact, which may reduce its credibility in practical applications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks at adversarial optimization in the context of adversarial robustness. Instead of framing the problem as a min-max optimization with respect to perturbation in input space, they consider perturbations in feature space, which allow to translate the min-max problem in a convex optimization algorithm with convergence guarantees. The authors later provide theoretical generalization bounds on this method, and provide numerical evidence that this method has advantages with respect to kernel ridge regression and standard robust training.\n\nSTRENGTHS AND WEAKNESSES:\n**Strenghts**\n\nThe paper discusses what i think to be a rather original approach to robust optimization. Furthermore, the paper is well written and presented with rigor.\n\n---\n\n**Weaknesses**\n\nThe Authors minimally elaborate on what is the exact set $\\Omega_{X}$ for different feature maps. They provide examples in Table 1, but I would in general stress that this connection is important to track the relevance of the the value $\\delta$ to express robustness (which generally considers perturbations in input space, not feature space)\n\nTo my understanding, these results are meaningful in the small dimensional setting, i.e. the scalings are for sufficiently large number of samples $n$, and the number of input dimensions $p$ is much smaller (see for example Table 2). This should also be a necessary assumption to have a direct connection between the fixed and random design (see line 153), as overfitting here becomes negligible. Can the authors remark on this point? In case I am correct, given the high popularity of high dimensional learning in the recent literature I would strongly suggest to remark this limitation in the revision.\n\nAt line 178 I found myself confused: the Authors mention \"optimal choices for $\\delta$\". To my understanding, $\\delta$ is a parameter that guarantees more robustness as it grows larger, and performances are in general monotonically decreasing as $\\delta$ increases (tradeoff between robustness and generalization). The narrative here seems to have a different angle, as $\\delta$ is framed more as a hyper-parameter that enables regularization. Can the authors explain this difference in perspective? This is maybe my main concern regarding the significance of this work. This is also not clear with respect to the results shown in Table 3, where $\\delta$ is directly associated with the resistance to some attack...\n\nIt is not totally clear to me how to parse the numerical experiments in the last section: in general, I would compare the test error of two algorithms for a fixed level of safety (which I identify as the resistance to adversarial perturbations). Here, a fixed value of $\\delta$ gives different robustness guarantees depending on the specific kernel (as shown for few examples in Table 1). It is possible that the proposed method has better generalization simply because it is guaranteeing \"less robustness\" for input perturbations (I am referring to Figure 3 in this case). Considering my previous comment, it is possible I did not fully understood what is the role of $\\delta$ the Authors want to stress here...\n\nQUESTIONS:\nLine 73-74: the Authors claim they consider square loss for simplicity and they point out that some developements can be applied more generally. Can they elaborate on this? Do not find an argument about this statement in the paper.\n\nLine 155: if the Authors refer to the fact that in ridge regression the optimal regularization $\\lambda$ depends on the noise, maybe adding one line of explanation, reference can improve the readability.\n\nLIMITATIONS:\nIt seems to me this paper does not have any theoretical insight for the high dimensional. If this is true, it is fair to discuss this.\n\nThe paper does not carry a clear connection between the meaning of $\\delta$ in input space and on a generic feature space. In words, a fixed value of $\\delta$ for a specific feature map, would provide robustness guarantees as if the optimization was like the one in Eq. (1) for which radius of $\\Omega_X$?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies adversarial kernel regression, where we aim at choosing the best function in an RKHS that is robust to worst-case perturbations in the input. The authors provide a relaxation of this problem by considering worst-case perturbations over the feature space, which allows the inner maximization to be solved in closed-form and leading to a convex minimization problem that can be solved with block-coordinate descent type algorithms. The authors also prove generalization bounds and demonstrate that unlike ridge regression, the optimal value of perturbation strength does not depend on noise level.\n\nSTRENGTHS AND WEAKNESSES:\n* Strengths:\nThe paper provides a novel and interesting trick to efficiently solve convex adversarial learning problems, and I believe there is some potential to use this insight in certain non-convex settings discussed below. The fact that the perturbation strength does not need to be adapted to the noise level makes this an appealing approach in settings where standard performance matters.\n\n* Weaknesses:\n    * In most use cases of adversarial training, one is interested in conditions of the type $\\Vert x \\Vert \\leq \\delta$ with a fixed $\\delta$ to ensure worst-case input perturbations can not degrade the model performance significantly. Since the relaxation provided in this paper may not be sharp, ensuring this constraint may require a very large perturbation budget in the feature space, which would eventually hurt standard performance.\n    * Some technical aspects of the paper can be improved. For example it is relatively standard to also study Rademacher-based generalization bounds in this setting, but the authors only consider a fixed-design excess risk which provides weaker guarantees and cannot rule out overfitting.\n    * While the authors measure the excess risk with respect to $f^\\*$, it might make more sense to compare the learned model's performance to the best model in the RKHS measured by population adversarial loss. $f^\\*$ may not be a good benchmark for adversarial robustness if it changes quickly with $x$.\n\nQUESTIONS:\n1. As the original adversarial training loss (with perturbations in the input space) is a maximization over convex functions, it is itself a convex minimization problem. How would solving the original problem compare to solving the relaxed version in the feature space?\n\n2. A relevant reference may be [1], where the authors study how to learn multi-index models in an adversarially robust manner with two-layer neural nets. The authors show that the first layer can be trained in a standard way, which leads to a data-adapted RKHS. In the second stage, they need to minimize the adversarial loss over this RKHS. The approach used in this paper can be used there to provide end-to-end guarantees for robustly learning multi-index models.\n\n3. What is the main technical difficulty in establishing random design generalization bounds? The Rademacher complexity for the adversarial loss has been studied extensively in prior works, so this seems within reach, and with this bound we can make a more confident comparison between adversarial learning and ridge regression.\n\n4. I couldn’t find the definition of the constant $R$ in the main text.\n\n5. How does it make sense to talk about $\\Vert d \\Vert_{\\mathcal{H}_j}$? Does this mean $d = \\sum_j d_j$, and the norm is $\\Vert d _j \\Vert _{\\mathcal{H}_j}$?\n\n---\n[1] A. Mousavi-Hosseini, A. Javanmard, M. A. Erdogdu. \"Robust Feature Learning for Multi-Index Models in High Dimensions.\" ICLR 2025.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an adversarial training framework used to kernel-based linear regression models, aiming to enhance model robustness. Traditional adversarial training methods typically formulate robustness as a min-max optimization problem in the input space and solve it using computationally intensive projected gradient descent (PGD). In contrast, this study introduces perturbations directly in the reproducing kernel Hilbert space (RKHS) feature space rather than the input space, enabling a convex reformulation of the problem that admits a closed-form solution. An optimization algorithm based on iterative kernel ridge regression is proposed. The paper establishes the convexity of the proposed formulation and analyzes its theoretical properties, deriving generalization bounds that guarantee its performance. Furthermore, the methodology is extended to multiple kernel learning, thereby increasing model expressiveness. Experimental evaluations on synthetic and real-world datasets demonstrate the superior performance of the proposed method compared to traditional kernel ridge regression approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n● Introduces perturbations in the RKHS feature space rather than the input space during adversarial training, presenting a closed-form solution for the inner-maximization problem, thus eliminating the need for PGD.\n● Provides theoretical insights by deriving upper bounds on the empirical L2 prediction error associated with the solutions obtained by the proposed algorithm, offering theoretical guarantees for performance.\n\n\nWeaknesses:\n\n● Kernel regression inherently involves computing the inverse of the kernel matrix, resulting in a high computational complexity of O(n³). Although acknowledged by the authors, this limitation is not addressed within the scope of the paper.\n● The proposed framework is limited to regression problems, lacking discussion or experimental validation for classification scenarios.\n● Due to methodological constraints, the diversity of experiments is limited. The datasets employed in experiments have relatively small numbers of features and data points, raising concerns about the scalability and numerical efficiency of the proposed method.\n● Additionally, the paper does not include a proof for Proposition 1, which is central to the theoretical foundation of the proposed method.\n\nQUESTIONS:\n● Can the proposed adversarial training framework be effectively scaled to handle larger datasets?\n● Is it possible to extend or adapt this method for classification tasks, and what modifications would be necessary?\n● How does the computational complexity of the proposed method compare with existing large-scale kernel methods?\n● Could alternative kernel approximation methods or matrix decomposition techniques address the computational complexity issue effectively?\n● Are there practical scenarios or applications where the feature-space perturbation provides significantly better robustness compared to input-space perturbations?\n● Are you perhaps confusing excess risk(used in line 150) with the L2 error? Excess risk refers to the difference between the risk of the empirical risk minimizer (ERM) and the risk of the Bayes estimator, and it is conceptually distinct from the L2 error.\n● In Table 3, what does the parameter p represent? In Appendix Table S.2, p appears to refer to the p-norm, whereas in Appendix E.1 and Table 2, p is used to denote the number of features in the data. This inconsistent notation is quite confusing. Moreover, the role of the p-norm is not clearly explained in the paper.\n● Traditional kernel regression already achieves robustness through the use of a ridge penalty, so it is unclear what additional benefits your method provides. Are there specific scenarios where standard kernel ridge regression is not robust, but your approach demonstrates robustness?\n\nLIMITATIONS:\n● The proposed framework does not address the high computational complexity of kernel regression, which involves O(n³) operations for kernel matrix inversion.\n● The methodology and experiments are restricted to regression problems, without considering or testing its generalization to classification tasks.\n● The experimental validations provided are limited to relatively small datasets, raising questions about the scalability and numerical efficiency of the approach.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper establishes the theoretical framework for kernel learning in the context of adversarial training and RKHS.
The proposed framework handles feature-space perturbation.
The efficient algorithm for solving the kernel adversarial training has been proposed, and generalization properties for the proposed framework are also provided.

STRENGTHS AND WEAKNESSES:
Strengths
1. This paper is well-written and solid.
3. This paper provides a new theoretical perspective of adversarial training from function spaces and non-parametric models.
4. The authors have considered different aspects of the proposed kernel adversarial training (formulation, numerical optimization, generalization bounds, multiple kernel learning, etc.)


Weakness
1. It will be better if the authors can explain how the generalization bound of the kernel adversarial training relies on the sample size n under different settings of the upper bound of adversarial error and compare this with kernel ridge regression and original adversarial training.
2. Some of the theorems/propositions in this paper are not self-contained. For example, what are R,\lambda, \sigma in Theorem 1 and Theorem 2? I would appreciate it if the authors could adjust them.
3. UpdateWeights in Algorithm 1 is confusing. Could the author write the details of this step?

QUESTIONS:
1. If \delta follows different orders of the sample size, then what is the order of the generalization bound?
2. How do the authors compare their results for kernel ridge regression in Theorem 2 with the rates developed in traditional statistics?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02