RAMP: Boosting Adversarial Robustness Against Multiple $l_p$ Perturbations for Universal Robustness

Authors: enyi  jiang
gagandeep  singh
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 03:38:07

Abstract: Most existing works focus on improving robustness against adversarial attacks bounded by a single $l_p$ norm using adversarial training (AT). However, these AT models' multiple-norm robustness (union accuracy) is still low, which is crucial since in the real-world an adversary is not necessarily bounded by a single norm. The tradeoffs among robustness against multiple $l_p$ perturbations and accuracy/robustness make obtaining good union and clean accuracy challenging. We design a logit pairing loss to improve the union accuracy by analyzing the tradeoffs from the lens of distribution shifts. We connect natural training (NT) with AT via gradient projection, to incorporate useful information from NT into AT, where we empirically and theoretically show it moderates the accuracy/robustness tradeoff. We propose a novel training framework \textbf{RAMP}, to boost the robustness against multiple $l_p$ perturbations. \textbf{RAMP} can be easily adapted for robust fine-tuning and full AT. For robust fine-tuning, \textbf{RAMP} obtains a union accuracy up to $53.3\%$ on CIFAR-10, and $29.1\%$ on ImageNet. For training from scratch, \textbf{RAMP} achieves a union accuracy of $44.6\%$ and good clean accuracy of $81.2\%$ on ResNet-18 against AutoAttack on CIFAR-10. Beyond multi-norm robustness \textbf{RAMP}-trained models achieve superior \textit{universal robustness}, effectively generalizing against a range of unseen adversaries and natural corruptions.

Main paper: 
 introduction: 
 Though deep neural networks (DNNs) demonstrate superior performance in various vision applications, they are vulnerable against adversarial examples [Goodfellow et al., 2014 , Kurakin et al., 2018] . Adversarial training (AT) [Tramèr et al., 2017 , Madry et al., 2017] which works by injecting adversarial examples into training for enhanced robustness, is currently the most popular defense. However, most AT methods address only a single type of perturbation [Wang et al., 2020 , Wu et al., 2020 , Carmon et al., 2019 , Gowal et al., 2020 , Raghunathan et al., 2020 , Zhang et al., 2021 , Debenedetti and Troncoso-EPFL, 2022 , Peng et al., 2023 , Wang et al., 2023 ]. An l ∞ robust model may not be robust against l p (p ̸ = ∞) attacks. Also, enhancing robustness against one perturbation type can sometimes increase vulnerability to others [Engstrom et al., 2017 , Schott et al., 2018] . On the contrary, training a model to be robust against multiple l p perturbations is crucial as it reflects real-world scenarios [Sharif et al., 2016 , Eykholt et al., 2018 , Song et al., 2018 , Athalye et al., 2018] where adversaries can use multiple l p perturbations. We show that multi-norm robustness is the key to improving generalization against other threat models [ Croce and Hein, 2022] . For instance, we show it enables robustness against perturbations not easily defined mathematically, such as image corruptions and unseen adversaries [Wong and Kolter, 2020] .Two main challenges exist for training models robust against multiple perturbations: (i) tradeoff among robustness against different perturbation models [Tramer and Boneh, 2019] and (ii) tradeoff between accuracy and robustness [Zhang et al., 2019 , Raghunathan et al., 2020] . Adversarial examples induce a shift from the original distribution, causing a drop in clean accuracy with AT [Xie et al., 2020 , Benz et al., 2021] . The distinct distributions created by l 1 , l 2 , l ∞ adversarial examples make the problem even more challenging. Through a finer analysis of the distribution shifts caused by these adversaries, we propose the RAMP framework to efficiently boost the Robustness Against Multiple Perturbations. RAMP can be used for both fine-tuning and training from scratch. It utilizes a novel logit pairing loss on a certain pair and connects NT with AT via gradient projection [Jiang et al., 2023] to improve union accuracy while maintaining good clean accuracy and training efficiency.Logit pairing loss. We visualize the changing of l 1 , l 2 , l ∞ robustness when fine-tuning a l ∞ -AT pre-trained model in Figure 1 using the CIFAR-10 training dataset. The DNN loses substantial robustness against l ∞ attack after only 1 epoch of fine-tuning: l 1 fine-tuning and E-AT [Croce and Hein, 2022] (red and yellow histograms under Linf category) both lose significant l ∞ robustness (compared with blue histogram under Linf category). Inspired by this observation, we devise a new logit pairing loss for a l q -l r tradeoff pair to attain better union accuracy, which enforces the logit distributions of l q and l r adversarial examples to be close, specifically on the correctly classified l q subsets. In comparison, our method (green histogram under Linf and union categories) preserves more l ∞ and union robustness than others after 1 epoch. We show this technique works on larger models and datasets (Section 5.1). RAMP preserves more l ∞ and union robustness.Connect natural training (NT) with AT. We explore the connections between NT and AT to obtain a better accuracy/robustness tradeoff. We find that NT can help with adversarial robustness: useful information in natural distribution can be extracted and leveraged to achieve better robustness. To this end, we compare the similarities of model updates of NT and AT layer-wise for each epoch, where we find and incorporate useful NT components into AT via gradient projection (GP), as outlined in Algorithm 2. In Figure 2 and Section 5.1, we empirically and theoretically show this technique strikes a better balance between accuracy and robustness, for both single and multiple l p perturbations. We provide a theoretical analysis of why GP works for adversarial robustness in Theorem A.2 & 4.5.• We design a new logit pairing loss to mitigate the l q -l r tradeoff for better union accuracy, by enforcing the logit distributions of l q and l r adversarial examples to be close. • We empirically and theoretically show that connecting NT with AT via gradient projection better balances the accuracy/robustness tradeoff for l p perturbations, compared with standard AT. • RAMP achieves good union accuracy, accuracy-robustness tradeoff, and generalizes better to diverse perturbations and corruptions (Section 5.1) achieving superior universal robustness (75.5% for common corruption and 26.1% union accuracy against unseen adversaries). RAMP fine-tuned DNNs achieve union accuracy up to 53.3% on CIFAR-10, and 29.1% on ImageNet. RAMP achieves a 44.6% union accuracy and good clean accuracy on ResNet-18 against AutoAttack on CIFAR-10. Our code is available at https://github.com/uiuc-focal-lab/RAMP. 
related work: 
 Adversarial training (AT). Adversarial Training (AT) usually employs gradient descent to discover adversarial examples, incorporating them into training for enhanced adversarial robustness [Tramèr et al., 2017 , Madry et al., 2017] . Numerous works focus on improving robustness by exploring the trade-off between robustness and accuracy [Zhang et al., 2019 , Wang et al., 2020] , instance reweighting [Zhang et al., 2021] , loss landscapes [Wu et al., 2020] , wider/larger architectures [Gowal et al., 2020, Debenedetti and Troncoso-EPFL, 2022] , data augmentation [Carmon et al., 2019 , Raghunathan et al., 2020] , and using synthetic data [Peng et al., 2023 , Wang et al., 2023] . However, these methods often yield DNNs robust against a single perturbation type while remaining vulnerable to other types.Robustness against multiple perturbations . Tramer and Boneh [2019] , Kang et al. [2019] observe that robustness against l p attacks does not necessarily transfer to other l q attacks (q ̸ = p). Previous studies [Tramer and Boneh, 2019 , Maini et al., 2020 , Madaan et al., 2021 , Croce and Hein, 2022] modified Adversarial Training (AT) to enhance robustness against multiple l p attacks, employing average-case [Tramer and Boneh, 2019] , worst-case [Tramer and Boneh, 2019, Maini et al., 2020] , and random-sampled [Madaan et al., 2021, Croce and Hein, 2022] defenses. There are also works [Nandy et al., 2020 , Liu et al., 2020 , Xu et al., 2021 , Xiao et al., 2022 , Maini et al., 2022] using preprocessing, ensemble methods, mixture of experts, and stability analysis to solve this problem. Ensemble models and preprocessing methods are weakened since their performance heavily relies on correctly classifying or detecting various types of adversarial examples. In certified training, Banerjee et al.[2024], Banerjee and Singh [2024] propose verification/certifiable training methods under different threat models for l p universal adversarial perturbation. However, prior works are hard to scale to larger models and datasets, e.g. ImageNet, due to the efficiency issue. Furthermore, Croce and Hein [2022] devise Extreme norm Adversarial Training (E-AT) and fine-tune a l p robust model on another l q perturbation to quickly make a DNN robust against multiple l p attacks. However, E-AT does not adapt to varying epsilon values. Our work demonstrates that the suboptimal tradeoff observed in prior studies can be improved with our proposed framework.Logit pairing in adversarial training. Adversarial logit pairing methods encourage logits for pairs of examples to be similar [Kannan et al., 2018 , Engstrom et al., 2018] . People apply this technique to both clean images and their adversarial counterparts, to devise a stronger form of adversarial training.In our work, we devise a novel logit pairing loss to train a DNN originally robust against l p attack to become robust against another l q (q ̸ = p) attack on the correctly predicted l p subsets, which helps gain better union accuracy.Adversarial versus distributional robustness. Sinha et al. [2018] theoretically studies the AT problem through distributional robust optimization. Mehrabi et al. [2021] establishes a pareto-optimal tradeoff between standard and adversarial risks by perturbing the test distribution. Other works explore the connection between natural and adversarial distribution shifts [Moayeri et al., 2022 , Alhamoud et al., 2023] , assessing transferability and generalizability of adversarial robustness across datasets. However, little research delves into distribution shifts induced by l 1 , l 2 , l ∞ adversarial examples and their interplay with the robustness-accuracy tradeoff [Zhang et al., 2019 , Yang et al., 2020 , Rade and Moosavi-Dezfooli, 2021] . Our work, inspired by recent domain adaptation techniques [Jiang, 2023 , Jiang et al., 2023] , designs a logit pairing loss and utilizes model updates from NT via GP to enhance adversarial robustness. We show that GP adapts to both single and multi-norm scenarios. 
at against multiple perturbations: 
 We consider a standard classification task with samples {(x i , y i )} N i=0 from an empirical data distribution D n ; we have input images x ∈ R d and corresponding labels y ∈ R k . Standard training aims to obtain a classifier f parameterized by θ to minimize a loss function [Madry et al., 2017 , Tramèr et al., 2017] aims to find a DNN robust against adversarial examples. It is framed as a min-max problem where a DNN is optimized using the worst-case examples within an adversarial region around each x i . Different types of adversarial regions B p (x, ϵ p ) = {x ′ ∈ R d : ∥x ′ -x∥ p ≤ ϵ p } can be defined around a given image x using various l p -based perturbations. Formally, we can write the optimization problem of AT against a certain l p attack as follows:L : R k × R k → R on D n . Adversarial training (AT)min θ E (x,y)∼ Dn max x ′ ∈Bp(x,ϵp) L(f (x ′ ), y)The above optimization is only for certain p values and is usually vulnerable to other perturbation types. To this end, prior works have proposed several approaches to train the network robust against multiple perturbations (l 1 , l 2 , l ∞ ) at the same time. We focus on the union threat model∆ = B 1 (x, ϵ 1 ) ∪ B 2 (x, ϵ 2 ) ∪ B ∞ (x, ϵ ∞ )which requires the DNN to be robust within the l 1 , l 2 , l ∞ adversarial regions simultaneously [Croce and Hein, 2022] . Union accuracy is then defined as the robustness against ∆ (i) for each x i sampled from D. In this paper, similar to the prior works, we use union accuracy as the main metric to evaluate the multiple-norm robustness. Apart from that, we define universal robustness as the generalization ability against a range of unseen adversaries and common corruptions. Specifically, we have average accuracy across five severity levels for common corruption and union accuracy against a range of unseen adversaries used in Laidlaw et al. [2020] .Worst-case defense follows the following min-max optimization problem to train DNNs using the worst-case example from the l 1 , l 2 , l ∞ adversarial regions:min θ E (x,y)∼ Dn max p∈{1,2,∞} max x ′ ∈Bp(x,ϵp) L(f (x ′ ), y)MAX [Tramer and Boneh, 2019] and MSD [Maini et al., 2020] fall into this category. Finding worst-case examples yields a good union accuracy but results in a loss of clean accuracy as the distribution of generated examples is different from the clean data distribution.Average-case defense train DNNs using the average of the l 1 , l 2 , l ∞ worst-case examples:min θ E (x,y)∼ Dn E p∈{1,2,∞} max x ′ ∈Bp(x,ϵp) L(f (x ′ ), y)AVG [Tramer and Boneh, 2019] is of this type. This method generally leads to good clean accuracy but suboptimal union accuracy as it does not penalize worst-case behavior within the l 1 , l 2 , l ∞ regions.Random-sampled defense. The defenses mentioned above lead to a high training cost as they compute multiple attacks for each sample. SAT [Madaan et al., 2021 ] and E-AT [Croce and Hein, 2022] randomly sample one attack out of each type at a time, contributing to a similar computational cost as standard AT on a single perturbation model. They achieve a slightly better union accuracy compared with AVG and relatively good clean accuracy. However, they are not better than worst-case defenses for multiple-norm robustness, since they do not consider the strongest attack within the union region all the time. 
ramp: 
 There are two main tradeoffs in achieving better union accuracy while maintaining good accuracy: 1. Among perturbations: there is a tradeoff among different attacks, e.g., a l ∞ pre-trained AT DNN is not robust against l 1 , l 2 perturbations, which makes the union accuracy harder to attain. Also, we observe there exists a main tradeoff pair of two attacks among the union over l 1 , l 2 , l ∞ attacks. 2. Accuracy and robustness: all defenses lead to degraded clean accuracy. To address these tradeoffs, we study the problem from the lens of distribution shifts.Interpreting tradeoffs from the lens of distribution shifts. The adversarial examples with respect to an empirical data distribution D n , adversarial region B p (x, ϵ p ), and DNN f θ generate a new adversarial distribution D a with samples {(x ′ i , y i )} N i=0 , that are correlated by adding certain perturbations but different from the original D n . Because of the shifts between D n and D a , DNN decreases performance on D n when we move away from it and towards D a . Also, the distinct distributions created by multiple perturbations, D l1 a , D l2 a , D l∞ a , contribute to the tradeoff among l 1 , l 2 , l ∞ attacks. To address the tradeoff among perturbations while maintaining good efficiency, we focus on the distributional interconnections between D n and D l1 a , D l2 a , D l∞ a . From the insights we get from above, we propose our framework RAMP, which includes (i) logit pairing to improve tradeoffs among multiple perturbations, and (ii) identifying and combining the useful DNN components using the model updates from NT and AT, to obtain a better robustness/accuracy tradeoff.Identify the Key Tradeoff Pair. We study the common case with l p norms ϵ 1 = 12, ϵ 2 = 0.5, ϵ ∞ = 8 255 on CIFAR-10 [Tramer and Boneh, 2019] . The distributions generated by the two strongest attacks show the largest shifts from D n ; also, they have the largest distribution shifts between each other because of larger and most distinct search areas. Thus, by comparing the single norm robustness of l p adversarially trained models, we select the two l p -AT models with the lowest l p robustness against themselves as the key tradeoff pair. They refer to the strongest attack since their l p robustness is low. The attack with the highest l p robustness is mostly included by the convex hull of the other two stronger attacks [Croce and Hein, 2022 ]. Here we identify l ∞ -l 1 as the key tradeoff pair.Figure 1 : Finetuning a l q -AT model on l r examples reduces l q robustness. To get a finer analysis of the l ∞ -l 1 tradeoff mentioned above, we visualize the changing of l 1 , l 2 , l ∞ robustness of the training dataset when we fine-tune a l ∞ pre-trained model with l 1 examples for 1 epochs, as shown in Figure 1: x-axis represents the robustness against different attacks and y-axis is the accuracy. After 1 epoch of finetuning on l 1 examples or performing E-AT, we lose much l ∞ robustness since blue/yellow histograms are much lower than the red histogram under the Linf category. RAMP preserves both l ∞ and union robustness more effectively: the green histogram is higher than the red/yellow histogram under Linf and Union categories. Specifically, RAMP maintains 14%, 28% more union robustness than E-AT and l 1 fine-tuning. The above observations indicate the necessity of preserving more l q robustness as we adversarially fine-tune with l r adversarial examples on a l q pre-trained AT model, with l q -l r as the key tradeoff pair, which inspires us to design our loss design with logit pairing. We want to enforce the union predictions between l q and l r (q ̸ = r) attacks: bringing the predictions of l q and l r (q ̸ = r) close to each other, specifically on the correctly predicted l q subsets. Based on our observations, we design a new logit pairing loss to enforce a DNN robust against one l q attack to be robust against another l r (q ̸ = r) attack.Enforcing the Union Prediction via Logit Pairing. The l q -l r (q ̸ = r) tradeoff leads us to the following principle to improve union accuracy: for a given set of images, when we have a DNN robust against some l q examples, we want it to be robust against l r examples as well. This serves as the main insight for our loss design: we want to enforce the logits predicted by l q and l r adversarial examples to be close, specifically on the correctly predicted l q subsets. To accomplish this, we design a KL-divergence (KL) loss between the predictions from l q and l r perturbations. For each batch of data (x, y) ∼ D, we generate l q and l r adversarial examples x ′ q , x ′ r and their predictions p q , p r using APGD [Croce and Hein, 2020] . Then, we select indices γ, which part elements of p q correctly predicts the ground truth y. We denote the size of the indices as n c , and the batch size as N . We compute a KL-divergence loss over this set of samples using KL(p q [γ]∥p r [γ]) (Eq. 1). For the subset indexed by γ, we want to push its l r logit distribution towards its l q logit distribution, such that we prevent losing more l q robustness when training with l r adversarial examples.EQUATIONTo further boost the union accuracy, apart from the KL loss, we add another loss term using a MAX-style approach in Eq. 2: we find the worst-case example between l q and l r adversarial regions by selecting the example with the higher loss. L max is a cross-entropy loss over the approximated worst-case adversarial examples. Here, we use L ce to represent the cross-entropy loss. Our final loss L combines L KL and L max , via a hyper-parameter λ in Eq. 3.L max = 1 N N i=0 max p∈{q,r} max x ′ i ∈Bp(x,ϵp) L ce (f (x ′ i ), y i ) (2) L = L max + λ • L KL (3)Algorithm 1 shows the pseudocode of robust fine-tuning with RAMP that leverages logit pairing.To improve the robustness and accuracy tradeoff against multiple perturbations, we explore the connections between AT and NT. Since extracting valuable information in NT aids in improving robustness (Section 4.2), we use gradient projection [Jiang et al., 2023] to compare and integrate natural and adversarial model updates, which yields an improved tradeoff between robustness and accuracy.NT can help adversarial robustness. Let us consider two models f 1 and f 2 , where f 1 is randomly initialized and f 2 undergoes NT on D n for k epochs: f 2 results in a better decision boundary and higher clean accuracy. Performing AT on f 1 and f 2 subsequently, intuitively, f 2 becomes more robust than f 1 due to its improved decision boundary, leading to fewer misclassifications of adversarial examples. This effect is empirically shown in Figure 2 . For AT (blue), standard AT against l ∞ attack [ Madry et al., 2017] is performed, while for AT-pre (red), 50 epochs of pre-training precede the standard AT procedure. AT-pre shows superior clean and robust accuracy on CIFAR-10 against l ∞ PGD-20 attack with ϵ ∞ = 0.031. Despite D n and D a are different, Figure 2 suggests valuable information in D n that potentially enhances performance on D a .AT with Gradient Projection. To connect NT with AT more effectively, we analyze the training procedures on D n and D a . We consider model updates over all samples from D n and D a , with the initial model f (r) at epoch r, and models f (r)n and f (r)a after 1 epoch of natural and adversarial training from the same starting point f (r) , respectively. Here, we compare the natural updates r) . Due to distribution shift, an angle exists between them. Our goal is to identify useful components from g n and incorporate them into g a for increased robustness in D a while maintaining accuracy in D n . Inspired by Jiang et al. [2023] , we layer-wisely compute the cosine similarity between g n and g a . For a specific layer l of g l n and g l a , we preserve a portion of g l n based on their cosine similarity score (Eq.4). Negative scores indicate that g l n is not beneficial for robustness in D a . Therefore, we filter components with similarity score ≤ 0. We define the GP (Gradient Projection) operation in Eq.5 by projecting g l a towards g l n .g n = f (r) n -f (r) and adversarial updates g a = f (r) a -f (cos(EQUATIONTherefore, the total projected (useful) model updates g p coming from g n could be computed as Eq. 6.We use M to denote all layers of the current model update. Note that l∈M concatenates all layers' useful natural model update components. A hyper-parameter β is used to balance the contributions of g GP and g a , as shown in Eq. 7. By finding a proper β (0.5 as in Figure 4c ), we can obtain better robustness on D a , as shown in Figure 2 and Figure 3 . In Figure 2 , with β = 0.5, AT-GP refers to AT with GP; for AT-GP-pre, we perform 50 epochs of NT before doing AT-GP. We see AT-GP obtains a better accuracy/robustness tradeoff than AT. We observe a similar trend for AT-GP-pre vs. AT-pre. Further, in Figure 3 , RN-18 l ∞ -GP achieves good clean accuracy and better robustness than RN-18 l ∞ against AutoAttack [Croce and Hein, 2020] .g p = l∈M GP( g l n , g l a ) (6) f (r+1) = f (r) + β • g p + (1 -β) • g a (7)Algorithm 1 Fine-tuning via Logit Pairing 1: Input: model f , input samples (x, y) from distribution D n , fine-tuning rounds R, hyper-parameter λ, adversarial regions B q , B r with size ϵ q and ϵ r , APGD attack. 2: for r = 1, 2, ..., R do x ′ q , p q ← APGD(B q (x, ϵ q ), y)5:x ′ r , p r ← APGD(B r (x, ϵ r ), y)6:γ ← where(argmax p q = y) r) 6: compute g p using Eq. 6 7: update f (r+1) using Eq. 7 with β and g a 8: end for 9: Output: model f .f n ← NT(f (r) , D) 4: f a ← AT(f (r) , B p , ϵ p , D) 5: compute g n ← f n -f (r) , g a ← f a - f (We define D n = {(x i , y i )} ∞ i=0 as the ideal data distribution with an infinite cardinality. Here, we consider a classifier f θ at epoch t. We define D a as the distribution created by {(x i +ϵ(f θ , x i , y i ), y i )} ∞ i=0 where (x i , y i ) ∼ D n . x i + ϵ(f θ , x i , y i ) denotes the perturbed image, which could be both single and multiple perturbations based on f θ itself.Assumption 4.1. We assume D n consists of N i.i.d. samples from the ideal distribution D n andD a = {(x i + ϵ(f θ , x i , y i ), y i )} N i=0 where (x i , y i ) ∼ D n consists of N i.i.d. samples from D a .We define the population loss as L D (θ) := E (x,y)∼D L(f (x), y), and let g D (θ) := ∇L D (θ). For simplification, we use g a := ∇L Da (θ), g a := ∇L Da (θ), andg n := ∇L Dn (θ). g GP = β • g p + (1 - β) • g a (Definition A.3) is the aggregation using GP. We define the following optimization problem. Definition 4.2 (Aggregation for NT and AT). f θ is trained by iteratively updating the parameterθ ← θ -µ • Aggr( g a , g n ),where µ is the step size. We seek an aggregation rule Aggr(•) = g Aggr such that after training, f θ minimizes the population loss function L Da (θ).We need g Aggr to be close to g a for each iteration, since g a is the optimal update on D a . Thus, we define L π -Norm and delta error to indicate the performance of different aggregation rules. Jiang, 2024] ). Given a distribution π on the parameter space θ, we define an inner product ⟨g D , gDefinition 4.3 (L π -Norm [EnyiD ′ ⟩ π = E θ∼π [⟨g D (θ), g D ′ (θ)⟩]. The inner product induces the L π - norm on g D as ∥g D ∥ π := E θ∼π ∥g D (θ)∥ 2 .We use L π -norm to measure the gradient differences under certain D. Definition 4.4 (Delta Error of an aggregation rule Aggr(•)). We define the following squared error term to measure the closeness between g Aggr and g a under D t a (distribution at time step t), i.e.,∆ 2 Aggr := E D t a ∥g a -g Aggr ∥ 2 π .Delta errors ∆ 2 AT and ∆ 2 GP measure the closesness of g GP , g a from g a in D a at each iteration. Theorem 4.5 (Error Analysis of GP). When the model dimension m → ∞, for an epoch t, we have an approximation of the error difference∆ 2 AT -∆ 2 GP as follows ∆ 2 AT -∆ 2 GP ≈ β(2 -β)E D t a ∥g a -g a ∥ 2 π -β 2 τ 2 ∥g a -g n ∥ 2 π τ 2 = E π [τ 2 ] ∈ [0, 1],where τ (θ) is the sin(•) value of the angle between g n and g a -g n .Theorem 4.5 shows ∆ 2 GP is generally smaller than ∆ 2 AT for a large model dimension during each iteration, as is the case for the models in our evaluation, with β = 0.5, since β(1 -β) > β 2 (0.75 > 0.25) and the small value of τ in practice (see Interpretation of Theorem A.2 in Appendix A, where we show the order of difference is between 1e -8 and 1e -12 ). Thus, GP achieves better robust accuracy than AT by achieving a smaller delta error; GP also obtains good clean accuracy by combining parts of the model updates from the clean distribution D n . Further, we provide an error analysis of a single gradient step in Theorem A.1 and convergence analysis in Theorem A.2, showing that a smaller Delta error results in better convergence. The full proof of all theorems is in Appendix A.We outline the AT-GP method in Algorithm 2 and it can be extended to the multiple-norm scenario. The overhead of this algorithm comes from natural training and GP operation. Their costs are small, and we discuss this more in Section 5.2. Combining logit pairing and gradient projection methods, we provide the RAMP framework which is similar to Algorithm 2, except that we replace line 4 of Algorithm 2 as Algorithm 1 line 3-9. 
experiment: 
 Datasets, baselines, and models. CIFAR-10 [ Krizhevsky et al., 2009] 2015]. We compare RAMP with following baselines: 1. SAT [Madaan et al., 2021] : randomly sample one of the l 1 , l 2 , l ∞ attacks. 2. AVG [Tramer and Boneh, 2019] : take the average of l 1 , l 2 , l ∞ examples. 3. MAX [Tramer and Boneh, 2019] : take the worst of l 1 , l 2 , l ∞ attacks. 4. MSD [Maini et al., 2020] : find the worst-case examples over l 1 , l 2 , l ∞ steepest descent directions during each step of inner maximization. 5. E-AT [Croce and Hein, 2022] : randomly sample between l 1 , l ∞ attacks. For models, we use PreAct- for CIFAR-10, as well as ResNet-50 and XCiT-S transformer for ImageNet.Implementations and Evaluation. For AT from scratch for CIFAR-10, we train PreAct ResNet-18 [He et al., 2016] with a lr = 0.05 for 70 epochs and 0.005 for 10 more epochs. We set λ = 2, β = 0.5 for training from scratch, and λ = 0.5 for robust fine-tuning. For all methods, we use 10 steps for the inner maximization in AT. For ImageNet, we perform 1 epoch of fine-tuning and use a learning rate lr = 0.005, λ = 0.5 for ResNet-50 and lr = 1e -4 , λ = 0.5 for XCiT-S models. We reduce the rate by a factor of 10 every 1 3 of the training epoch and set the weight decay to 1e -4 . We use APGD with 5 steps for l ∞ and l 2 , 15 steps for l 1 . Settings are similar to [Croce and Hein, 2022] . We use the standard values of ϵ 1 = 12, ϵ 2 = 0.5, ϵ ∞ = 8 255 for CIFAR-10 and ϵ 1 = 255, ϵ 2 = 2, ϵ ∞ = 4 255 for ImageNet. We focus on l ∞ -AT models for fine-tuning, as Croce and Hein [2022] shows their higher union accuracy for the ϵ values in our evaluation. We report the clean accuracy, robust accuracy against {l 1 , l 2 , l ∞ } attacks, union accuracy, universal robustness against common corruptions and unseen adversaries, as well as runtime for RAMP. The robust accuracy is evaluated using Autoattack [Croce and Hein, 2020] . More implementation details are in Appendix B. Robust fine-tuning. In Table 2 , we apply RAMP to larger models and datasets (ImageNet). However, the implementation of other baselines is not publicly available and Croce and Hein [2022] do not report other baseline results except E-AT on larger models and datasets, so we only compare against E-AT in Table 2 , which shows RAMP consistently obtains better union accuracy and accuracyrobustness tradeoff than E-AT. We observe that RAMP improves the performance more as the model becomes larger. We obtain the SOTA union accuracy of 53.3% on CIFAR-10 and 29.1% on ImageNet.RAMP with varying ϵ 1 , ϵ 2 , ϵ ∞ values. We provide results with 1. (ϵ 1 = 12, ϵ 2 = 0.5, ϵ ∞ = 2 255 ) where ϵ ∞ size is small and 2. (ϵ 1 = 12, ϵ 2 = 1.5, ϵ ∞ = 8 255 ) where ϵ 2 size is large, using PreAct ResNet-18 model for CIFAR-10 dataset: these cases have different tradeoff pair compared to Universal Robustness. In Table 4 , we report average accuracy against common corruptions and union accuracy against unseen adversaries from Laidlaw et al.[2020] (implementation details are in Appendix B.3). We compare against l p pretrained models, E-AT, MAX, winninghand [Diffenderfer et al., 2021] (a SOTA method for natural corruptions) using WideResNet-28-10 architecture on the CIFAR-10 dataset. Compared to E-AT and MAX, RAMP achieves 4% higher accuracy for common corruptions with five severity levels and 2-4% better union accuracy against multiple unseen adversaries. Winninghand has high corruption robustness but no adversarial robustness. The results show that RAMP obtains a better robustness and accuracy tradeoff with stronger universal robustness. In Appendix B.3, we evaluate on ResNet-18 to support this fact further.Sensitivities of λ. We perform experiments with different λ values in [0.1, 0.5, 1.0, 1.5, 2, 3, 4, 5] for robust fine-tuning and [1.5, 2, 3, 4, 5, 6] for AT from scratch using PreAct-ResNet-18 model for CIFAR-10 dataset. In Figure 4 , we observe a decreased clean accuracy when λ becomes larger. We pick λ = 2.0 for training from scratch (Figure 4a ) and λ = 0.5 for robust fine-tuning (Figure 4b ) in our main experiments, as these values of λ yield both good clean and union accuracy.Choices of β. Figure 4c shows the performance of RAMP with varying β values on CIFAR-10 ResNet-18 experiments. We pick β = 0.5 for combining natural training and AT via GP, which achieves comparatively good robustness and clean accuracy. This choice is also based on Theorem 4.5 when β(2 -β) has the largest difference from β 2 (0.75 vs 0.25). Table 5 : RAMP with l ∞ , l 1 , l 2 -RN-18-AT models on CIFAR-10 with standard epsilons.Fine-tune l p AT models with RAMP. Table 5 shows the robust fine-tuning results using RAMP with l ∞ -AT (q = ∞, r = 1), l 1 -AT (q = 1, r = ∞), l 2 -AT (q = ∞, r = 1) RN-18 models for CIFAR-10 dataset.For l ∞ -l 1 tradeoffs, RAMP on l ∞ -AT pre-trained model achieves the best union accuracy.Computational analysis and Limitations. The extra training costs of AT-GP are small, e.g. for each epoch on ResNet-18, the extra NT takes 6 seconds and the standard AT takes 78 seconds using a single NVIDIA A100 GPU, and the GP operation only takes 0.04 seconds on average. RAMP is more expensive than E-AT and less expensive than MAX. We have a complete runtime analysis in Appendix B.2. We notice occasional drops in clean accuracy during fine-tuning with RAMP. In some cases, union accuracy improves slightly but clean accuracy and single l p robustness reduce. Further, we find no negative societal impact from this work. 
conclusion: 
 We introduce RAMP, a framework enhancing multiple-norm robustness and achieving superior universal robustness against corruptions and perturbations by addressing tradeoffs among l p perturbations and accuracy/robustness. We  
 
 Appendix: A.1 Proof of Theorem A.2We first show what happens during one step of optimization, where we highlight the importance of analyzing delta error. Theorem A.1. Consider model parameter θ ∼ π and an aggregation rule Aggr(•) with step size µ > 0. Define the updated parameter as θ + := θ -µ g Aggr (θ).Assuming the gradient ∇L(θ) is γ-Lipschitz in θ for any input, and let the step size µ ≤ 1 γ , we haveE Da,θ [L Da (θ + ) -L Da (θ)] ≤ -µ 2 (∥g a ∥ 2 π -∆ 2 Aggr ).Proof. The proof is the same as Theorem A.1 in [Enyi Jiang, 2024] .Theorem A.2 (Convergence of Aggr(•)). For any probability measure π over the parameter space, and an aggregation rule Aggr(•) with step size µ > 0. We update the parameter for T steps by θ t+1 := θ t -µ g Aggr (θ t ). Assume the gradient ∇L(θ) and g Aggr (θ) are γ 2 -Lipschitz in θ such that θ t → θ Aggr . ∆ Aggr_max is the Delta error at time t ′ when ∥ g Aggr ( θ Aggr ) -∇L D t ′ a ( θ Aggr )∥ 2 is maximized. Then, given step size µ ≤ 1 γ and a small enough ϵ > 0, with probability at least 1 -δ we have∥∇L D T a (θ T )∥ 2 ≤ 1 δ 2 C ϵ • ∆ 2 Aggr_max + O(ϵ) 2 + O 1 T + O(ϵ),whereC ϵ = E D t ′ a [1/π(B ϵ ( θ Aggr ))] 2 and B ϵ ( θ Aggr ) ⊂ R mis the ball with radius ϵ centered at θ Aggr . The C ϵ measures how well π covers where the optimization goes.Proof. Denote random function f : R m → R + asEQUATIONwhere the randomness comes from D a . Note that f is γ-Lipschitz by assumption. Now we consider B ϵ ( θ Aggr ) ⊂ R m , i.e., the ball with radius ϵ centered at θ Aggr . Then, by γ-Lipschitzness we haveE θ∼π f (θ) = f (θ) dπ(θ) ≥ Bϵ( θAggr) ( f ( θ Aggr ) -γϵ) dπ(θ) = ( f ( θ Aggr ) -γϵ)π(B ϵ ( θ Aggr )) Therefore, f ( θ Aggr ) ≤ 1 π(B ϵ ( θ Aggr )) • E θ∼π f (θ) + O(ϵ).Taking expectation w.r.t. D a on both sides, we haveE Da f ( θ Aggr ) ≤ E Da 1 π(B ϵ ( θ Aggr )) • E θ∼π f (θ) + O(ϵ) ≤ E Da 1 π(B ϵ ( θ Aggr )) 2 • E Da E θ∼π f (θ) 2 + O(ϵ) (Cauchy-Schwarz) = C ϵ • E Da E θ∼π f (θ) 2 + O(ϵ) (by definition of C ϵ ) ≤ C ϵ • E Da E θ∼π f (θ) 2 + O(ϵ) (Jensen's inequality) = C ϵ • ∆ 2 Aggr + O(ϵ)By Markov's inequality, with probability at least 1 -δ we have a sampled dataset D a such thatf ( θ Aggr ) ≤ 1 δ E Da f ( θ Aggr ) ≤ 1 δ C ϵ • ∆ 2 Aggr + O(ϵ/δ) (9)Conditioned on such event, we proceed on to the optimization part.Note that Theorem A.1 characterizes how the optimization works for one gradient update. We denote D t a as the data distribution D a at time step t. Therefore, for any time step t = 0, . . . , T -1, we can apply Theorem A.1 which only requires the Lipschitz assumption:L D t a (θ t+1 ) -L D t a (θ t ) ≤ - µ 2 ∥∇L D t a (θ t )∥ 2 -∥ g Aggr (θ t ) -∇L D t a (θ t )∥ 2 .We notice that D t a changes based on θs of different time steps. On both sides, to sum over t = 0, . . . , T -1, we first consider two terms:(L D t a (θ t+1 ) -L D t a (θ t )) + (L D t-1 a (θ t ) -L D t-1 a (θ t-1 ))To compare L D t a (θ t ) and L D t-1 a (θ t ), since D t a optimizes one more step than D t-1 a , we assumeL D t a (θ t ) ≤ L D t-1a (θ t ) for ∀t. Therefore, we have:(L D t a (θ t+1 ) -L D t a (θ t )) + (L D t-1 a (θ t ) -L D t-1 a (θ t-1 )) ≥ (L D t a (θ t+1 ) -L D t-1 a (θ t )) + (L D t-1 a (θ t ) -L D t-1 a (θ t-1 ))Summing up all time steps,L D t a (θ t+1 ) -L D 0 a (θ 0 ) ≤ L D t a (θ t+1 ) -L D t-1 a (θ t ) + L D t-1 a (θ t ) -L D t-2 a (θ t-1 ) + ... -L D 0 a (θ 0 ) ≤ - µ 2 T -1 t=0 ∥∇L D t-1 a (θ t )∥ 2 - T -1 t=0 ∥ g Aggr (θ t ) -∇L D t a (θ t )∥ 2 .Dividing both sides by T , and with regular algebraic manipulation we derive1 T T -1 t=0 ∥∇L D t-1 a (θ t )∥ 2 ≤ 2 µT (L D 0 a (θ 0 ) -L D T -1 a (θ T )) + 1 T T -1 t=0 ∥ g Aggr (θ t ) -∇L D t a (θ t )∥ 2 .Note that we assume the loss function L D (θ) := E (x,y)∼D L(f (x), y), is non-negative. Thus, we haveEQUATION)Note that we assume given D a we have θ t → θ Aggr . Therefore, for any ϵ > 0 there exist T ϵ such thatEQUATION)This implies that ∀t > T ϵ :EQUATION)Moreover, (11) also implies ∀t 1 , t 2 > T ϵ :EQUATION)Now, let's get back to (10). For ∀T > T ϵ we have1 T T -1 t=0 ∥∇L D t-1 a (θ t )∥ 2 ≤ 2L D 0 a (θ 0 ) µT + 1 T Tϵ-1 t=0 ∥ g Aggr (θ t ) -∇L D t a (θ t )∥ 2 + 1 T T -1 t=Tϵ ∥ g Aggr (θ t ) -∇L D t a (θ t )∥ 2 = O 1 T + 1 T T -1 t=Tϵ ∥ g Aggr (θ t ) -∇L D t a (θ t )∥ 2 = O 1 T + 1 T T -1 t=Tϵ ∥ g Aggr (θ t ) -g Aggr ( θ Aggr ) + g Aggr ( θ Aggr ) -∇L D t a (θ t )∥ 2 ≤ O 1 T + 1 T T -1 t=Tϵ ∥ g Aggr (θ t ) -g Aggr ( θ Aggr )∥ + ∥ g Aggr ( θ Aggr ) -∇L D t a (θ t )∥ 2 (triangle inequality) = O 1 T + 1 T T -1 t=Tϵ O(ϵ) + ∥ g Aggr ( θ Aggr ) -∇L D t a (θ t )∥ 2 (by (12)) = O 1 T + O(ϵ) + 1 T T -1 t=Tϵ ∥ g Aggr ( θ Aggr ) -∇L D t a ( θ Aggr ) + ∇L D t a ( θ Aggr ) -∇L D t a (θ t )∥ ≤ O 1 T + O(ϵ) + 1 T T -1 t=Tϵ ∥ g Aggr ( θ Aggr ) -∇L D t a ( θ Aggr )∥ + O(ϵ) 2(by ( 13))EQUATIONEquation 14 bounds the left hand side with the maximum ∥ g Aggr ( θ Aggr ) -∇L D t a ( θ Aggr )∥ 2 one can get during the optimization steps. Here, we assume at time t ′ , the largest value is attained. We denote ∆ 2Aggr_max as the delta error at time step t ′ . Then, we can continue with what we have done at the beginning of the proof of this theorem:(14) = O 1 T + O(ϵ) + f ( θ Aggr ) 2 (by (8)) ≤ O 1 T + O(ϵ) + 1 δ C ϵ • ∆ 2 Aggr_max + O(ϵ/δ) 2 (by (9))Therefore, combining the above we finally have: for ∀T > T ϵ with probability at least 1 -δ,EQUATIONTo complete the proof, let us investigate the left-hand side.1 T T -1 t=0 ∥∇L D t-1 a (θ t )∥ 2 = 1 T Tϵ-1 t=0 ∥∇L D t a (θ t )∥ 2 + 1 T T -1 t=Tϵ ∥∇L D t a (θ t )∥ 2 = O 1 T + 1 T T -1 t=Tϵ ∥∇L D t a (θ t )∥ 2 ≥ O 1 T + 1 T T -1 t=Tϵ ∥∇L D t a (θ t ) -∇L D T a (θ T )∥ -∥∇L D T a (θ T )∥ 2 (triangle inequality) = O 1 T + 1 T T -1 t=Tϵ O(ϵ) + ∥∇L D T a (θ T )∥ 2(by ( 13))EQUATION)Combining ( 15) and ( 16), we finally have∥∇L D T a (θ T )∥ 2 ≤ O 1 T + O(ϵ) + 1 δ 2 C ϵ • ∆ 2 Aggr_max + O(ϵ) 2 ,which completes the proof.A.2 Proof of Theorem 4.5To prove Theorem 4.5, we first use the following definitions and lemmas from [Enyi Jiang, 2024] , to get the delta errors of Gradient Projection (GP) and standard adversarial training (AT): Definition A.3 (GP Aggregation). Let β ∈ [0, 1] be the weight that balances between g a and g n . The GP aggregation operation isGP ( g a , g n ) = (1 -β) g a + βProj + ( g a | g n ) .where Proj + ( g a | g n ) = max{⟨ g a , g n ⟩, 0} g n /∥ g n ∥ 2 is the operation that projects g a to the positive direction of g n . Definition A.4 (AT Aggregation). The AT aggregation operation isAT ( g a ) = g a .standard AT only leverages the gradient update on D a .Lemma A.5 (Delta Error of GP). Given distributions D a , D a and D n , as well as the model updates g a , g a , g n on these distributions per epoch, we have ∆ 2 GP as follows∆ 2 GP ≈ (1 -β) 2 + 2β -β 2 m E Da ∥g a -g a ∥ 2 π + β 2 τ 2 ∥g a -g n ∥ 2 π ,In the above equation, m is the model dimension andτ 2 = E π [τ 2 ] ∈ [0, 1] where τ (θ) is the sin(•)value of the angle between g n and g a -g n . ∥ • ∥ π is the π-norm over the model parameter space.Proof. The proof is the same as Theorem 4.4 in Enyi Jiang [2024] .Lemma A.6 (Delta Error of AT). Given distributions D a , D a and D n , as well as the model updates g a , g a , g n on these distributions per epoch, we have ∆ 2 AT as follows∆ 2 AT = E Da ∥g a -g a ∥ 2 π ,where ∥ • ∥ π is the π-norm over the model parameter space.Then, we prove Theorem 4.5.Theorem A.7 (Error Analysis of GP). When the model dimension is large (m → ∞) at time step t, we have∆ 2 AT -∆ 2 GP ≈ β(2 -β)E D t a ∥g a -g a ∥ 2 π -β 2 τ 2 ∥g a -g n ∥ 2 π . τ 2 = E π [τ 2 ] ∈ [0, 1]where τ is the sin(•) value of the angle between g n and g a -g n , ∥ • ∥ π is the π-norm over the model parameter space.Proof.∆ 2 AT -∆ 2 GP ≈ E D t a ∥g a -g a ∥ 2 π -(1 -β) 2 + 2β-β 2 m E Da ∥g a -g a ∥ 2 π -β 2 τ 2 ∥g a -g n ∥ 2 π = 1 -((1 -β) 2 + 2β-β 2 m ) E D t a ∥g a -g a ∥ 2 π -β 2 τ 2 ∥g a -g n ∥ 2 π = (1 + 1 m )β(2 -β)E D t a ∥g a -g a ∥ 2 π -β 2 τ 2 ∥g a -g n ∥ 2 πWhen m → ∞, we have a simplified version of the error difference as follows∆ 2 AT -∆ 2 GP ≈ β(2 -β)E D t a ∥g a -g a ∥ 2 π -β 2 τ 2 ∥g a -g n ∥ 2 π Interpretation. When β = 0.5, we can usually show ∆ 2 AT > ∆ 2 GP , because β(2 -β) > β 2 τ 2 (0.75 > 0.25)for the coefficients of two terms. We estimate the actual values of terms E D a t ∥g a -g a ∥ 2 π (variance), ∥g a -g n ∥ 2 π (bias), and τ using the estimation methods in Enyi Jiang [2024] . Table 6 displays the values of those terms as well as the error differences on ResNet18 experiments at epoch 5, 10, 15, 20, 60. We plot the changing of these terms on the ResNet18 experiment in Figure 5 . The order of difference is always positive and usually smaller than 1e -08 and approaches the order of 1e -12 in the end. In this section, we provide more training details, additional experiment results on the universal robustness of RAMP to common corruptions and unseen adversaries, runtime analysis of RAMP, additional ablation studies on different logit pairing losses, and AT from random initialization results on CIFAR-10 using WideResNet-28-10.We set the batch size to 128 for the experiments on ResNet-18 and WideResNet-28-10 architectures.We use an SGD optimizer with 0.9 momentum and 5e -4 weight decay. For other experiments on ImageNet, we use a batch size of 64 to fit into the GPU memory for larger models. We present runtime analysis results demonstrating the fact that RAMP is more expensive than E-AT and less expensive than MAX in Table 7 . These results, recorded in seconds per epoch, were obtained using a single A100 40GB GPU. RAMP consistently supports that fact in all experiments. In this section, we show RAMP can generalize better to other corruptions and unseen adversaries on union accuracy for stronger universal robustness.Implementations. For the l 0 attack, we use Croce et al. [2022] with an epsilon of 9 pixels and 5k query points. For common corruptions, we directly use the implementation of RobustBench [Croce et al., 2020] for evaluation across 5 severity levels on all corruption types used in Hendrycks and Dietterich [2019] . For other unseen adversaries, we follow the implementation of Laidlaw et al. [2020] , where we set eps = 12 for the fog attack, eps = 0.5 for the snow attack, eps = 60 for the gabor attack, eps = 0.125 for the elastic attack, and eps = 0.125 for the jpeglinf attack with 100 iterations. For ResNet-18 experiments, we do not compare with Winninghand [Diffenderfer et al., 2021] since it uses a Wide-ResNet architecture. Also, we select the strongest baselines (E-AT and MAX) from the Wide-ResNet experiment results to compare for ResNet-18 experiments on universal robustness.Results. For the ResNet-18 training from scratch experiment on CIFAR-10, in Table 8 and 9, we also show RAMP generally outperforms by 0.5% on common corruptions and 7% on union accuracy against unseen adversaries compared with E-AT. [6, 9, 12, 15] , and ϵ 2 = [0.25, 0.75, 1.0, 1.5]. We provide additional RAMP results compared with related baselines with training from scratch and performing robust fine-tuning in Section B.4.1 and Section B.4.2, respectively. We observe that RAMP can surpass E-AT with significant margins as well as a better accuracy-robustness tradeoff for both training from scratch and robust fine-tuning with λ = 2.0 for training from scratch and λ = 0.5 for robust fine-tuning in most cases.Changing l ∞ perturbations with ϵ ∞ = [ 2 255 , 4 255 , 12 255 , 16 255 ]. Table 10 and Table 11 show that RAMP consistently outperforms E-AT [Croce and Hein, 2022] on union accuracy when training from scratch. [6, 9, 12, 15] . Table 12 and Table 13 show that RAMP consistently outperforms E-AT [Croce and Hein, 2022] on union accuracy when training from scratch.Changing l 2 perturbations with ϵ 2 = [0.25, 0.75, 1.0, 1.5]. Table 14 and Table 15 show that RAMP consistently outperforms E-AT [Croce and Hein, 2022] on union accuracy when training from scratch. 17 show that RAMP consistently outperforms E-AT [Croce and Hein, 2022] on union accuracy when performing robust fine-tuning. [6, 9, 12, 15] . Table 12 and Table 13 show that RAMP consistently outperforms E-AT [Croce and Hein, 2022] on union accuracy when performing robust fine-tuning.Changing l 2 perturbations with ϵ 2 = [0.25, 0.75, 1.0, 1.5]. Table 14 and Table 15 show that RAMP consistently outperforms E-AT [Croce and Hein, 2022] on union accuracy when performing robust fine-tuning. In this section, we test RAMP with robust fine-tuning using two more different logit pairing losses:(1) Mean Squared Error Loss (L mse ) (Eq. 17), (2) Cosine-Similarity Loss (L cos ) (Eq. 18). We replace the KL loss we used in the paper using the following losses. We use the same lambda value λ = 1.5 for both cases.EQUATIONTable 22 displays RAMP robust fine-tuning results of different logit pairing losses using PreAct-ResNet-18 on CIFAR-10 with λ = 1.5. We see those losses generally improve union accuracy compared with baselines in Table 24 . L cos has a better clean accuracy yet slightly worsened union accuracy. L mse has the best union accuracy and the worst clean accuracy. L KL is in the middle of the two others. However, we acknowledge the possibility that each logit pairing loss may have its own best-tuned λ value. Implementations. We use a cyclic learning rate with a maximum rate of 0.1 for 30 epochs and adopt the outer minimization trades loss from Zhang et al. [2019] with the default hyperparameters, same as Croce and Hein [2022] ; also, we set λ = 2.0 and β = 0.5 for training RAMP. Additionally, we use the WideResNet-28-10 architecture same as Zagoruyko and Komodakis [2016] for our reimplementations on CIFAR-10.Results. Since the implementation of experiments on WideResNet-28-10 in Croce and Hein [2022] paper is not public at present, we report our implementation results on E-AT, where our results show that RAMP outperforms E-AT in union accuracy with a significant margin, as shown in Table 23 . Also, we experiment with using the trade loss (RAMP w trades) for the outer minimization, we observe that RAMP w trades achieves a better union accuracy at the loss of some clean accuracy. Implementations. For robust fine-tuning with ResNet-18, we perform 3 epochs on CIFAR-10. We set the learning rate as 0.05 for PreAct-ResNet-18 and 0.01 for other models. We set λ = 0.5 in this case. Also, we reduce the learning rate by a factor of 10 after completing each epoch.Table 24 shows the robust fine-tuning results using PreAct ResNet-18 model on the CIFAR-10 dataset with different methods. The results for all baselines are directly from the E-AT paper [Croce and Hein, 2022] where the authors reimplemented other baselines (e.g., MSD, MAX) to achieve better union accuracy than presented in the original works. RAMP surpasses all other methods on union accuracy. 81.1 ± 0.2 45.4 ± 0.3 66.1 ± 0.2 47.2 ± 0.1 43.1 ± 0.2 + RAMP (λ = 0.5) 81.5 ± 0.1 45.5 ± 0.2 66.4 ± 0.2 47.0 ± 0.1 42.9 ± 0.2In Table 25 , we apply robust fine-tuning on the PreAct ResNet-18 model for the CIFAR-10 dataset with 5, 7, 10, 15 epochs, and compare it with E-AT. RAMP consistently outperforms the baseline on union accuracy, with a larger improvement when we increase the number of epochs.In this section, we provide additional t-SNE visualizations of the multiple-norm tradeoff and robust fine-tuning procedures using different methods. C.1 Pre-trained l 1 , l 2 , l ∞ AT modelsFigure 6 shows the robust accuracy of l 1 , l 2 , l ∞ AT models against their respect l 1 , l 2 , l ∞ perturbations, on CIFAR-10 using PreAct-ResNet-18 architecture. Similar to Figure ??, l ∞ -AT model has a low l 1 robustness and vice versa. In this common choice of epsilons, we further confirm that l ∞ -l 1 is the key trade-off pair.We provide the complete visualizations of robust fine-tuning for 3 epochs on CIFAR-10 using l 1 examples, E-AT, and RAMP. Rows in l 1 fine-tuning (Figure 7 ), E-AT fine-tuning (Figure 8 ), and RAMP fine-tuning (Figure 9 ) show the robust accuracy against l ∞ , l 1 , l 2 attacks individually, of epoch 0, 1, 2, 3, respectively. We observe that throughout the procedure, RAMP manages to maintain more l ∞ robustness during the fine-tuning with more points colored in cyan, in comparison with two other methods. This visualization confirms that after we identify a l p -l r (p ̸ = r) key tradeoff pair, RAMP successfully preserves more l p robustness when training with some l r examples via enforcing union predictions with the logit pairing loss.Figure 6 : l 1 , l 2 , l ∞ pre-trained RN18 l ∞ -AT models with correct/incorrect predictions against l 1 , l 2 , l ∞ attacks. Correct predictions are colored with cyan and incorrect with magenta. Each row represents l ∞ , l 1 , l 2 AT models, respectively. Each column shows the accuracy concerning a certain l p attack.  


Decision: This paper studies the question of jointly achieving robustness against multiple adversarial perturbations. The main idea is to use a logit-pairing loss that makes the logits achieved on multiple perturbations match each other. Then, the authors used gradient projection to further align the updates from adversarial examples and ordinary non-adversarial examples.

Two reviewers are concerned about the paper. For example, one reviewer believes that logit pairing sounds like a “standard” idea and lacks novelty. Furthermore, focusing on the L1, L2, and L-inf attacks are not representative enough, and more studies on other types of perturbations should be considered, such as common corruptions, which the authors have addressed during the rebuttal. There is another concern that the paper does not provide precise characterizations of adversarial distributions for different norms.

Two other reviewers favored accepting the paper due to the solid theoretical and empirical results. One reviewer even championed the paper despite the diverging scores. 

IMHO, the problem of studying the tradeoff between different norms is legitimate because that’s a good starting point for the community to go beyond a single perturbation norm. For example, it’s quite intuitive that L2 and L-infinity attacks will lead to different distributions of adversarial examples, and training against one type of perturbation can lead to a distributional shift on the other. I think this is a common problem, and focusing on the L1, L2, and L-infinity is probably due to the convention in this field and the consideration of tractability. 

The reviewers provided some good suggestions for improving the paper. For example, instead of showing a T-SNE plot in Figure 1, one can just show a line plot. After discussing with the reviewer, the authors showed that RAMP achieves an improved accuracy-robustness tradeoff than a previous baseline, which convinced one reviewer.

In summary, I recommend accepting the paper despite the diverging scores.

Reviews: review: paper_summary: paper_summary: The paper introduces a new addition to the adversarial training literature -- it proposes to augment standard adversarial training [Madry et al.] with a specific logits pairing loss and, the existing technique, of gradient project. 

The paper shows strong results on the union of lp-norm adversarial accuracy (over l-1, l-inf and l-2 using the standard CIFAR-10 dataset and standard perturbation radii).
strength_weakness: strength_weakness: The one main weakness is that Table 1 must be re-generated using all the test set points in CIFAR-10, not just 1000.

Presentation-wise, Figure 1 is very confusing; there is no need to plot t-SNEs of image when one wants to simply show classification labels. Please consider re-plotting this figure (with a line plot, histogram, etc.).

The writing, overall, is ok, but could also be improved: for example, lines 162-167, are approximately repeated from earlier on in the paper.
questions: questions: See above.
limitations: limitations: The authors have adequately addressed the limitations of their work.
paper_summary: paper_summary: This work proposes a training framework RAMP, to boost the robustness against multiple $\ell_p$ perturbations, through connecting NT with AT via gradient projection.
strength_weakness: strength_weakness: My main concern is why the distributions of adversarial examples generated using $\ell_1$, $\ell_2$, and $\ell_\infty$ norms are distinct?
Is this due to inconsistent (unreasonable) radius settings across different $\ell_p$-norms?
What constitutes a reasonable radius setting for various norms?
These norms can be bounded by each other theoretically, what is the essential meaning to study this problem?
questions: questions: ref weakness
limitations: limitations: As I mentioned before, in my opinion, I think this work needs to discuss more about the essential meaning of the problem.
paper_summary: paper_summary: The paper proposes a framework called RAMP to boost the adversarial robustness against multiple $l_p$ perturbations. Compared with existing methods, RAMP can better retain the robustness against the original type of perturbations when fine-tuning the model. The authors provide theoretical and empirical justifications to demonstrate the effectiveness of RAMP.
strength_weakness: strength_weakness: Despite the strength in the last section, I have the following concerns:

1. Novelty in the algorithm: logit pairing is not a new idea. Actually, in Equation (1), if we replace $p_r$ with the probability of the clean input, the algorithm will become TRADES. Some findings, such as trade-offs between accuracy and robustness, are not new.

2. Generality of the algorithm: the manuscript only studies the joint robustness among $l_1$, $l_2$ and $l_\infty$ perturbations, especially the $l_1$ and $l_\infty$ ones. This is too limited. The authors should consider more generic cases, such as $l_0$ bounded perturbations and common corruptions (Gaussian noise, color re-mapping e.t.c.). I believe this will make the algorithm more generally applicable.

3. Novelty and precision of the theorem: the theoretical tools employed to analyze convergence in Theorem 4.5 are standard and not unique. In Theorem 4.6, if $\beta = 0$, $\widehat{g}\_a$ should be similar to $g_{GP}$, so $\Delta_{AT}$ should be very close to $\Delta_{GP}$. Furthermore, the left hand side of the approximation should be close to $0$, can the author explain how the right hand side approximation reflects this? In addition, using $\simeq$ is not rigorous, the authors should at least point out the order of the difference between the left hand side and the right hand side.
questions: questions: The weakness part points out the major questions I would ask. Below are additional relatively minor questions:

1. Why don't use AutoAttack as the evaluation metric in Figure 2?

2. In Figure 4, what is the difference between MAX and RAMP with $\lambda = 0$?

My major concern of this paper is about the novelty, because both the motivation (logit pairing) and many parts of the analysis are standard. In addition, the algorithm is not generic, as it is only evaluated under $l_1$, $l_2$ and $l_\infty$ bounded perturbations. Therefore, I cannot recommend acceptance at this point. I welcome the authors to address my concerns in the rebuttal.
limitations: limitations: The limitations and the societal impact are briefly discussed in the end of the paper.
paper_summary: paper_summary: This paper addresses the challenge of training Machine Learning (ML) models that show good robustness against multiple l_p attacks and accuracy. This problem has already been studied in previous work [33,6], but the papers discuss the problem from the lens of distribution shift. This point of view represents the starting point for proposing a new adversarial training (AT) algorithm called RAMP based on two building blocks: (i) a new logit pairing loss that allows a model robust against a l_p norm to exhibit high robustness w.r.t. another l_q norm (with q \neq p); (ii) a new model parameter update based on the combinations of model updates from natural training and adversarial training to train a model that observes a better accuracy/union robustness tradeoff that SOTA AT models. A deep theoretical analysis shows that, for large models and a reasonable choice of parameter values, the novel parameter update allows the adversarial training algorithm to converge faster, training a model with higher robustness and accuracy. The extensive experimental evaluation highlights that the trained large models on ImageNet and Cifar-10 exhibit a better accuracy-union robustness tradeoff than SOTA AT proposals [6].
strength_weakness: strength_weakness: **Analysis of time per epoch is not opportunely deepened (addressed by the authors)**: the analysis regarding the computational time per epoch is limited since the results are shown for only one model (ResNet-18) on only one dataset (CIFAR-10), but seven other models are employed in the experimental evaluation. Since a lot of additional results have been included in the Appendix, I would have expected to see also more results supporting the statement ``` RAMP has around two times the runtime cost of E-AT and 2/3 cost of MAX``` (Section 5.2). For example, the authors may want to report the time per epoch on AT the ResNet-18 and the WRN-28-10 from scratch on CIFAR-10 (as reported in [6]) and on training another model on ImageNet.

**Trade-off between robustness and accuracy seems to not be properly addressed**: Even though RAMP allows the model to obtain better union accuracy, the clean accuracy is in any case lower than the clean accuracy obtained by using the baseline E-AT [6]. As an example, the RAMP fine-tuned RN-50-$\ell_\infty$ shows a clean accuracy that is 3.8 points lower than the one obtained by the E-AT fine-tuned model and a gain in union accuracy that is one point (Table 1). Moreover, more than half of the results shown in the Appendix and some results in the main body show that the gain in the union accuracy is obtained at the expense of a comparable loss in the clean accuracy. If the trade-off between robustness and accuracy had been properly addressed, I would have expected to see a comparable clean accuracy and better union accuracy than the one obtained by E-AT. These results limit the novelty of RAMP since it seems to be slower than E-AT and only allows the model to obtain better union accuracy without properly addressing the accuracy/robustness trade-off. I suggest that the authors explain why these cases happen and how to improve also the accuracy of the obtained model.
questions: questions: - Can the authors show more results supporting the fact that the time per epoch required by RAMP is partially longer than the time per epoch required by E-AT? (See the Weaknesses section for more details)

- Is it possible to use RAMP to obtain an accuracy close to the one obtained by using E-AT and a better union accuracy? (See the Weaknesses section for more details)

**Updates after authors' response**: The authors provided additional results to address the first question, effectively addressing the first weakness. After discussion, the answer to the second question is more convincing. RAMP seems to address the accuracy-robustness trade-off better than E-AT even by choosing small values of $\lambda$, as the authors wrote. So, choosing a small value of $\lambda$ helps preserve the accuracy while still outperforming the models trained with E-AT regarding union accuracy. This choice could be right in general. 

Reasons for Not Giving a Higher Score: RAMP does not allow the user to obtain an impressive gain in union accuracy as E-AT did when the corresponding paper was published. However, it allows the model to obtain a few points more of union accuracy while preserving an accuracy similar to that of models trained with E-AT. I can say that this paper will have a moderate to high impact, but I am not convinced that it will have a high impact for sure.

Reasons for Not Giving a Lower Score: In my opinion, the work is solid from a theoretical and methodological point of view. The experimental results are more convincing after the discussion with the authors.
limitations: limitations: The authors have explained the limitations in Section 5.2 in brief. However, the limitations should be addressed better:
- The analysis of the computational cost should be supported with more experimental results.
- The authors should better substantiate the sentence ``` We notice occasional drops in clean accuracy during fine-tuning with RAMP```. Indeed, the drop in accuracy is exhibited also when the model is trained from scratch (see Section B.2.1 in the Appendix). Moreover, the gain in the union accuracy is often comparable to the loss in clean accuracy. Thus, the accuracy/robustness trade-off seems not to be properly addressed.

This work has no negative societal and ethical impacts since it contributes to improving the robustness of AI systems in practice. The experimental and implementation details have been documented.

Comments: title: Thank you and we provide a further explanation
Further response on correlations among adversarial distributions
Thank you and we provide additional results
response
Official Comment by Reviewer Qhq5
Thanks for the additional results you are providing
Thank you and response to choose lambda values
Thank you and more justifications on why l1, l2, linf threat models are important for real-world applications.
response
response to author
Thanks for your response, I have other concerns
follow up
follow up
follow up
follow up
comment: Thanks for your reply and here we provide a further explanation.

**Regarding the implementation**: For l0 attack, we use [1] with an epsilon of 9 pixels and 5k query points. For common corruptions, we directly use the implementation of robustbench [2] for evaluation across 5 severity levels on all corruption types used in [3]. For other unseen adversaries, we follow the implementation of [4], where we set eps=12 for fog attack, eps = 0.5 for snow attack, eps=60 for gabor attack, eps=0.125 for elastic attack, and eps=0.125 for jpeg linf attack (we have mentioned those epsilons in the previous table). We will update the implementation information in the next version of our paper.

We think it is standard to provide additional results and respond to reviewer requests at NeurIPS (see examples [4,5]), which does not necessarily require a resubmission. 

**Correlations among the distributions**. Please see our newest response to Reviewer 6TV5.

[1] https://github.com/fra31/sparse-rs/tree/master

[2] https://github.com/RobustBench/robustbench/tree/master/robustbench 

[3] Benchmarking Neural Network Robustness to Common Corruptions and Perturbations.

[4] https://github.com/cassidylaidlaw/perceptual-advex 

[5] https://openreview.net/forum?id=IkD1EWFF8c 

[6] https://openreview.net/forum?id=v5Aaxk4sSy
Thank you for your further questions and we respond to them as follows.

People have tried to theoretically characterize the adversarial distribution [1,2,3], but to the best of our knowledge, there have been no successful attempts at the precise characterizations of adversarial distributions for different norms theoretically. This is because the adversarial distributions change dynamically during training due to updates to model weights. Even after many years, people have not been able to characterize it for a single norm on DNNs. We believe that this problem itself is challenging and deserves a separate investigation.

Our goal is to achieve better robustness-accuracy tradeoff by providing insights into the relationship between natural and adversarial distributions (Theorem 4.6 and 4.7 in our paper). Our theory does not explicitly analyze the correlations among different norms, as we approach them collectively by treating the mixture as a single distribution. However, we offer insights into robustness-accuracy tradeoff under this framework, which applies to both individual and multiple attacks. This perspective provides a generalizable understanding that is relevant across various attack scenarios.

Further, we studied the correlations between adversarial distributions of multiple norms by identifying the key tradeoff pair. The distributions generated by the two strongest attacks show the largest shifts from the natural distribution; also, they have the largest distribution shifts between each other because of larger and most distinct search areas. Thus, by calculating the ball volume for each attack, we select the two with the largest volumes as the key tradeoff pair, which also informs our logit pairing method. Our current approach demonstrates effectiveness by improving union accuracy with significant margins. 

Also, we want to point out that we study the multiple-norm robustness problem not only because $l_1, l_2, l_\infty$ norms are commonly used. We show RAMP with adversarial training on these norms can generalize to a variety of other perturbations and corruptions, serving as the basis for many real-world attacks. This is supported by several results we provided in our previous rebuttal, illustrating their applicability and relevance in practical scenarios.

[1] Explicit Tradeoffs between Adversarial and Natural Distributional Robustness.

[2] Precise Tradeoffs in Adversarial Training for Linear Regression.

[3] Fundamental Tradeoffs in Distributionally Adversarial Training.
I thank the authors for the detailed responses.

1. Regarding the perturbation beyond $l_1$, $l_2$ and $l_\infty$ attacks, I believe including the tables presented in the rebuttal is beneficial. However, the authors should also provide detailed information about the implementation of $l_0$ attack, common corruption attack e.t.c. for readers to comprehensively assess the effectiveness of the proposed method. This needs a significant revision of the paper, and I believe it needs a re-submission to ensure a rigorous review process.

2. I agree with Reviewer 6TV5 that the correlation among the distributions caused by different types of perturbation is not clearly investigated in the theoretial analyses. In addition, the algorithmic difference between the proposed method and existing ones (such as TRADES) is not big. Therefore, I have concerns about the novelty of this work.

Based on the issues above, I regret to say that I cannot increase my rating based on the current manuscript.
Thank you very much for considering our work and for increasing the score. We will incorporate the updated results into the next version of our paper. In the meantime, we would like to share additional results that demonstrate improved tradeoffs with smaller $\lambda$ values compared to E-AT (training from scratch, $\lambda$ = 2).

|      Models      | Methods | Clean | linf |  l2  |  l1  | union |
|:----------------:|:-------:|:-----:|:----:|:----:|:----:|:-----:|
|     resnet-18    |   E-AT  |  82.2 | 42.7 | 67.5 | 53.6 |  42.4 |
|                  |   RAMP  |  81.9 | 46.1 | 66.6 | 48.9 |  44.4 |
| wideresnet-28-10 |   E-AT  |  79.2 | 44.2 | 64.9 | 54.9 |  44.0 |
|                  |   RAMP  |  84.5 | 45.5 | 67.3 | 47.8 |  44.0 |
Authors also mentioned $\ell_{1,2,\infty}$ are commmonly used in adversarial robustness research, thus they studied the mixture of them. Combining these norms seems more like an incremental work. The real question is: 
**What's the correlation between adversarial distributions generated by different norms? How RAMP deals with this correlation?** This work did not provide any theoretical analysis here. 

In this point, I agree with Reviewer 3Hj9, this work lacks novelty, just an incremental work.
Thanks also to you for the quick response. Since you have provided more evidence that the small values of $\lambda$ allow RAMP to outperform E-AT, as you previously wrote, I am more confident now that RAMP can address better than E-AT the accuracy-robustness trade-off. Thus, I will increase my score. I invite you to update the results in the next version of your paper.
Thanks for the quick response. The numbers are different because we regenerated both E-AT (we trained them again) and RAMP results testing on all testing data points instead of the 1000 points in the paper, as suggested by Reviewer jXrr (see our response to Reviewer jXrr).
I greatly appreciate your dedication in showing that RAMP can consistently outperform E-AT when training from scratch, even with a small $\lambda$, to be coherent with what you wrote. 

I have only one question about the data. I suppose you are producing the data in Table 1 again. Why are the results obtained with E-AT different from the results you show in Table 1 in the paper? Are you training the models again using E-AT to ensure that they are all trained in the same setting?
We thank the reviewer for their detailed reply. We will include the time cost analysis in the next version of the paper. Also, here we provide more results on $\lambda$ value experiments. RAMP can achieve a better robustness-accuracy tradeoff compared with E-AT by choosing consistent $\lambda$ values for multiple models. Here we set $\lambda=0.5$ for robust fine-tuning. Also, we are running from scratch experiments and will report as soon as we have the results.

|        Models        | Methods | Clean | linf |  l2  |  l1  |   union  |
|:--------------------:|:-------:|:-----:|:----:|:----:|:----:|:--------:|
|         RN-50        |   E-AT  |  85.3 | 46.5 | 68.3 | 45.3 |   41.6   |
|                      |   RAMP  |  84.3 | 47.0 | 67.7 | 46.5 | **43.3** |
|       WRN-34-20      |   E-AT  |  87.8 | 49.0 | 71.6 | 49.8 |   45.1   |
|                      |   RAMP  |  87.1 | 49.7 | 70.8 | 50.4 | **46.9** |
| WRN-28-10 (*) carmon |   E-AT  |  89.3 | 51.8 | 74.6 | 53.3 |   47.9   |
|                      |   RAMP  |  89.2 | 55.9 | 74.7 | 55.7 | **52.7** |
|  WRN-28-10 (*) gowal |   E-AT  |  89.8 | 54.4 | 76.1 | 56.0 |   50.5   |
|                      |   RAMP  |  89.4 | 55.9 | 74.7 | 56.0 | **52.9** |
|     WRN-70-16 (*)    |   E-AT  |  89.6 | 54.4 | 76.7 | 58.0 |   51.6   |
|                      |   RAMP  |  90.6 | 54.7 | 74.6 | 57.9 | **53.3** |
We thank the reviewer for the detailed reply and here we provide further justifications on why $l_1$, $l_2$, $l_\infty$ attacks are important for real-world attacks.

1. We would like to emphasize that the use of  $l_1$, $l_2$, and $l_\infty$ norms has been extensively adopted in adversarial learning research, primarily as a foundational approach to enhance robustness against well-defined and mathematically tractable threat models [1,2,3]. While these attack models may appear straightforward, the challenge of developing models that are genuinely robust against them at the same time remains a significant and unresolved issue. The critical importance of these simpler attack paradigms is further underscored by their central role in the most widely used adversarial toolkits, such as Foolbox [4], and in benchmarks like RobustBench [5], which predominantly focus on evaluating robustness against these norm-based attacks. Further, many physical real-world adversarial attacks [7,8,9,10] are based on these norms.

2. RAMP shows a better ability to generalize to more complex norms. For example, in the case of the Wasserstein adversarial attack [11], RAMP outperforms E-AT and MAX by 1-3% across different perturbation sizes, as demonstrated in the table below. Besides, we are happy to try other norms suggested by the reviewer if there are any.

|      |  eps = 0.001 |  eps = 0.003 |  eps = 0.005 |
|------|--------------|--------------|--------------|
| MAX  |         75.7 |         70.6 |         66.1 |
| E-AT |         76.6 |         71.8 |     67.7 |
| RAMP |     79.5 |     73.4 |     67.5 |

3. Techniques for enhancing  $l_1$,  $l_2$, and $l_\infty$ robustness can be extended to address perturbation sets that are not easily defined mathematically, such as image corruptions or adversarial lighting variations [6]. Our new results on generalizing to other perturbations and corruptions (see tables in response to reviewers 6TV5, 3Hj9, and Qhq5) confirm that point with the best robustness-accuracy tradeoff among $l_p$ pretrained, SOTA image corruption, E-AT, and MAX models. 

In conclusion, RAMP represents an advancement in enhancing union adversarial robustness and the robustness-accuracy tradeoff across norms/perturbations/corruptions, while maintaining a relatively good computational efficiency.




[1] Explaining and harnessing adversarial examples.

[2] Towards evaluating the robustness of neural networks.

[3] Towards Deep Learning Models Resistant to Adversarial Attacks. 

[4] https://github.com/bethgelab/foolbox/tree/master

[5] https://robustbench.github.io/ 

[6] Learning perturbation sets for robust machine learning.

[7] Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition.

[8] Robust Physical-World Attacks on Deep Learning Visual Classification.

[9] Physical Adversarial Examples for Object Detectors.

[10] Synthesizing Robust Adversarial Examples.

[11] Stronger and Faster Wasserstein Adversarial Attacks.
To avoid misleading the AC, I increase my score to 5, but I sit on the fence for this work.
I thank authors for their detailed response. 

**"In real-world settings, adversarial attacks often involve multiple types of perturbations, necessitating models to be robust against $\ell_1$, $\ell_2$, and $\ell_\infty$ attacks."**  
There are so many vector norms and matrix norms, why $\ell_1$, $\ell_2$, and $\ell_\infty$ vector norms are important in **real-world** adversarial attacks? 
While this work combines previously established norm-based attacks, it appears to be an incremental advancement rather than a fundamental exploration. 
The crucial question remains unanswered: What intrinsic properties of these three norms make them more suitable for **mixture** compared to other potential norms? A deeper investigation into this fundamental issue could provide valuable insights and potentially lead to more robust defense mechanisms.

**"The distributions are distinct because these norms measure distances differently"**  
Given that these norms can be bounded by one another, I believe it is more accurate to characterize their relationship as ''correlated'' rather than ''distinct''. Further exploration of this correlation could yield valuable insights.

**"$\epsilon_1$, $\epsilon_2$, $\epsilon_\infty$ in experiments"**  
While the corner cases (or extremal spaces) for different norms are indeed distinct, a critical question remains: What justifies the focus on these three specific norms ($\ell_1$, $\ell_2$, and $\ell_\infty$) rather than exploring corner cases for other norms?

All in all, while this work contributes to the field, it appears to be an incremental advancement. It would be better to explore the above questions. I sit on the fence for this work.
I thank the authors for their response.

The answer to the first question is clear, thanks to the additional results provided. I strongly encourage you to include these results in the next version of the paper.

The response to the second question is interesting. Noting that 'RAMP obtains a better robustness and accuracy tradeoff by generalizing better to common corruptions and other unseen adversaries' is certainly another valuable contribution. Additionally, I appreciate your emphasis on the importance of the $\lambda$ value in achieving the desired accuracy-robustness trade-off. However, I am somewhat concerned that $\lambda$ may need to be fine-tuned to reach the desired trade-off, as we lack clear evidence that the union accuracy will decrease as minimally as shown in Figure 4 when moving to lower $\lambda$ values. Indeed, that figure refers to only one model, and we have no data on the impact of $\lambda$ when training other models. Fine-tuning can require a lot of computational time, since you shown that training with RAMP may require more than twice the time required by E-AT. Moreover, why did you not consistently choose a smaller $\lambda$ value, even when training from scratch (you set $\lambda=5$ in the paper), to obtain accuracy comparable to E-AT while consistently achieving higher robustness?

To sum up, your analysis has improved my opinion of your work, but there is insufficient evidence regarding the influence of the $\lambda$ value on clean and union accuracy across multiple models to say that the accuracy-robustness trade-off is properly addressed.
Dear reviewer,

We want to thank you again for your efforts in reviewing our submission.

We have tried our best to respond to your questions. If our responses address your concerns, we sincerely hope you can reconsider the scores. We will also be very happy to answer any follow-up questions. Look forward to your updates.

Thanks,
Authors
Dear reviewer,

We want to thank you again for your efforts in reviewing our submission.

We have tried our best to respond to your questions. If our responses address your concerns, we sincerely hope you can reconsider the scores. We will also be very happy to answer any follow-up questions. Look forward to your updates.

Thanks,
Authors
Dear reviewer,

We want to thank you again for your efforts in reviewing our submission.

We have tried our best to respond to your questions. If our responses address your concerns, we sincerely hope you can reconsider the scores. We will also be very happy to answer any follow-up questions. Look forward to your updates.

Thanks,
Authors
Dear reviewer,

We want to thank you again for your efforts in reviewing our submission.

We have tried our best to respond to your questions. If our responses address your concerns, we sincerely hope you can reconsider the scores. We will also be very happy to answer any follow-up questions. Look forward to your updates.

Thanks,
Authors