Title: On Proper Learnability between Average- and Worst-case Robustness
Venue: NeurIPS 2023
Accepted: yes
Keywords: Adversarial Robustness

PAC Learning
Abstract: Recently, Montasser at al. (2019) showed that finite VC dimension is not sufficient for proper adversarially robust PAC learning. In light of this hardness, there is a growing effort to study what type of relaxations to the adversarially robust PAC learning setup can enable proper learnability. In this work, we initiate the study of proper learning under relaxations of the worst-case robust loss. We give a family of robust loss relaxations under which VC classes are properly PAC learnable with sample complexity close to what one would require in the standard PAC learning setup. On the other hand, we show that for an existing and natural relaxation of the worst-case robust loss, finite VC dimension is not sufficient for proper learning. Lastly, we give new generalization guarantees for the adversarially robust empirical risk minimizer.

On Proper Learnability between Average- and Worst-case Robustness

Abstract

Recently, Montasser et al. [2019] showed that finite VC dimension is not sufficient for proper adversarially robust PAC learning. In light of this hardness, there is a growing effort to study what type of relaxations to the adversarially robust PAC learning setup can enable proper learnability. In this work, we initiate the study of proper learning under relaxations of the adversarially robust loss. We give a family of robust loss relaxations under which VC classes are properly PAC learnable with sample complexity close to what one would require in the standard PAC learning setup. On the other hand, we show that for an existing and natural relaxation of the adversarially robust loss, finite VC dimension is not sufficient for proper learning. Lastly, we give new generalization guarantees for the adversarially robust empirical risk minimizer.

1 Introduction

As deep neural networks become increasingly ubiquitous, their susceptibility to test-time adversarial attacks has become more and more apparent. Designing learning algorithms that are robust to these test-time adversarial perturbations has garnered increasing attention by machine learning researchers and practitioners alike. Prior work on adversarially robust learning has mainly focused on learnability under the worst-case adversarially robust risk [Montasser et al., 2019, Attias et al., 2021, Cullina et al., 2018],

/negationslash

where U ( x ) ⊂ X is an arbitrary but fixed perturbation set (for example /lscript p balls). In practice, worstcase robustness is commonly achieved via Empirical Risk Minimization (ERM) of the adversarially robust loss or some convex surrogate [Madry et al., 2017, Wong and Kolter, 2018, Raghunathan et al., 2018, Bao et al., 2020]. However, a seminal result by Montasser et al. [2019] shows that any proper learning rule, including ERM, even when trained on an arbitrarily large number of samples, may not return a classifier with small adversarially robust risk. These high generalization gaps for the adversarially robust loss have also been observed in practice [Schmidt et al., 2018]. Even worse, empirical studies have shown that classifiers trained to achieve worst-case adversarial robustness exhibit degraded nominal performance [Dobriban et al., 2020, Raghunathan et al., 2019, Su et al., 2018, Tsipras et al., 2018, Yang et al., 2020, Zhang et al., 2019, Robey et al., 2022].

In light of these difficulties, there has been a recent push to study when proper learning, and more specifically, when learning via ERM is possible for achieving adversarial robustness. The ability to achieve test-time robustness via proper learning rules is important from a practical standpoint. It

aligns better with the current approaches used in practice (e.g. (S)GD-trained deep nets), and proper learning algorithms are often simpler to implement than improper ones. In this vain, Ashtiani et al. [2022] and Bhattacharjee et al. [2022] consider adversarial robust learning in the tolerant setting, where the error of the learner is compared with the best achievable error with respect to a slightly larger perturbation set. They show that the sample complexity of tolerant robust learning can be significantly lower than the current known sample complexity for adversarially robust learning and that proper learning via ERM can be possible under certain assumptions. Additionally, Ashtiani et al. [2020] provide some sufficient conditions under which proper robust learnability of VC classes becomes possible under a PAC-type framework of semi-supervised learning. More recently, Attias et al. [2022] show that finite VC dimension is sufficient for proper semi-supervised adversarially robust PAC learning if the support of the marginal distribution on instances is known. On the other hand, Attias and Hanneke [2023] study the adversarially robust PAC learnability of real-valued functions and show that every convex function class is properly learnable.

/negationslash

In a different direction, several works have studied the consistency of various surrogates of the adversarially robust loss /lscript U ( h, ( x, y )) = sup z ∈ U ( x ) { h ( z ) = y } [Awasthi et al., 2022a,b, 2023, Mao et al., 2023], while others have considered relaxing its worst-case nature [Robey et al., 2022, Li et al., 2020, 2021, Laidlaw and Feizi, 2019, Rice et al., 2021]. However, the PAC learnability of these relaxed notions of the adversarially robust loss has not been well-studied.

In this paper, we study relaxations of the worst-case adversarially robust learning setup from a learning-theoretic standpoint. We classify existing relaxations of worst-case adversarially robust learning into two approaches: one based on relaxing the loss function and the other based on relaxing the benchmark competitor. Much of the existing learning theory work studying relaxations of adversarial robustness focus on the latter approach. These works answer the question of whether proper PAC learning is feasible if the learner is evaluated against a stronger notion of robustness. In contrast, we focus on the former relaxation and pose the question: can proper PAC learning be feasible if we relax the adversarially robust loss function itself? In answering this question, we make the following main contributions:

We show that the finiteness of the VC dimension is not sufficient for properly learning a natural relaxation of the adversarially robust loss proposed by Robey et al. [2022]. Our proof techniques involve constructing a VC class that is not properly learnable.

We give a family of adversarially robust loss relaxations that interpolate between averageand worst-case robustness. For these losses, we use Rademacher complexity arguments relying on the Ledoux-Talagrand contraction to show that all VC classes are learnable via ERM.

We extend a property implicitly appearing in margin theory (e.g., see Mohri et al. [2018, Section 5.4]), which we term 'Sandwich Uniform Convergence' (SUC), to show new generalization guarantees for the adversarially robust empirical risk minimizer.

2 Preliminaries and Notation

Throughout this paper we let [ k ] denote the set of integers { 1 , ..., k } , X denote an instance space, Y = { -1 , 1 } denote our label space, and D be any distribution over X × Y . Let H ⊆ Y X denote a hypothesis class mapping examples in X to labels in Y .

2.1 Problem Setting

In the standard adversarially robust learning setting, the learner, during training time, picks a set G ⊆ X X 2 of perturbation functions g : X → X against which they wish to be robust. At test time, an adversary intercepts the labeled example ( x, y ) , exhaustively searches over the perturbation set to find the worst g ∈ G , and then passes the perturbed instance g ( x ) to the learner. The learner makes a prediction ˆ y and then suffers the loss { ˆ y = y } . The goal of the learner is to find a hypothesis h ∈ Y X that minimizes the adversarially robust risk R G ( h ; D ) = E ( x,y ) ∼ D [ /lscript G ( h, ( x, y ))] where /lscript G ( h, ( x, y )) := sup g ∈ G { h ( g ( x )) = y } is the adversarially robust loss.

/negationslash

/negationslash

In practice, however, such a worst-case adversary may be too strong and unnatural, especially in high-dimension. Accordingly, we relax this model by considering a lazy , computationally-bounded adversary that is unable to exhaustively search over G , but can randomly sample a perturbation function g ∼ µ given access to a measure µ over G . In this setup, the learner picks both a perturbation set G and a measure µ over G . At test-time, the lazy adversary intercepts the labeled example ( x, y ) , randomly samples a perturbation function g ∼ µ , and then passes the perturbed instance g ( x ) to the learner. From this perspective, the goal of the learner is to output a hypothesis such that the probability that the lazy adversary succeeds in sampling a bad perturbation function, for any labeled example in the support of D , is small. In other words, the learner strives to be robust to most, but not all, perturbation functions in G .

We highlight that the set G and the measure µ are fixed and chosen by the learner at training time. As a result, the measure µ does not depend on the unperturbed point x . Allowing the measure to depend on the unperturbed point x is an interesting future direction that lies between our model and adversarial robustness. Nevertheless, fixing the measure µ is still relevant from a practical standpoint and non-trivial from a theoretical standpoint. Indeed, a popular choice for G and µ are /lscript p balls and the uniform measure respectively. These choices have been shown experimentally to strike a better balance between robustness and nominal performance [Robey et al., 2022]. Moreover, in Section 3, we show that even when the measure µ is fixed apriori, there are learning problems for which ERM does not always work. This parallels the hardness result from Montasser et al. [2019], who prove an analogous result for adversarially robust PAC learning.

2.2 Probabilistically Robust Losses and Proper Learnability

/negationslash

Given a perturbation set G and measure µ , we quantify the probabilistic robustness of a hypothesis h on a labeled example ( x, y ) by considering losses that are a function of P g ∼ µ [ h ( g ( x )) = y ] . For a labeled example ( x, y ) ∈ X × Y , g ∼ µ [ h ( g ( x )) = y ] measures the fraction of perturbations in G for which the classifier h is non-robust. Observe that g ∼ µ [ h ( g ( x )) = y ] = 1 -y g ∼ µ [ h ( g ( x ))] 2 is an affine transformation of quantity y g ∼ µ [ h ( g ( x ))] , the probabilistically robust margin of h on ( x, y ) with respect to ( G , µ ) . Thus, we focus on loss functions that operate over the margin y g ∼ µ [ h ( g ( x ))] . One important loss function is the ρ -probabilistically robust loss,

/negationslash

/negationslash

where ρ ∈ [0 , 1) is selected apriori. The ρ -probabilistically robust loss was first introduced by Robey et al. [2022] for the case when X = d , g c ( x ) = x + c , and the set of perturbations G = { g c : c ∈ ∆ } for some ∆ ⊂ d . In this paper, we generalize this loss to an arbitrary instance space X and perturbation set G . As highlighted by Robey et al. [2022], this notion of robustness is desirable as it nicely interpolates between worst- and average-case robustness via an interpretable parameter ρ , while being more computationally tractable compared to existing relaxations.

In this work, we are primarily interested in understanding whether probabilistic relaxations of the adversarially robust loss enable proper learning . That is, given a hypothesis class H , perturbation set and measure ( G , µ ) , loss function /lscript G ,µ ( h, ( x, y )) = /lscript ( y g ∼ µ [ h ( g ( x ))]) , and labeled samples from an unknown distribution D , our goal is to design a learning algorithm A : ( X × Y ) ∗ → H such that for any distribution D over X × Y , the algorithm A , given a sample of labeled examples from D , finds a hypothesis h ∈ H with low risk with regards to /lscript G ,µ ( h, ( x, y )) .

Definition 1 (Proper Probabilistically Robust PAC Learnability) . Let /lscript G ,µ ( h, ( x, y )) denote an arbitrary probabilistically robust loss function. For any /epsilon1 , δ ∈ (0 , 1) , the sample complexity of probabilistically robust ( /epsilon1 , δ ) -learning H with respect to /lscript G ,µ , denoted n ( /epsilon1 , δ ; H , /lscript G ,µ ) is the smallest number m ∈ for which there exists a proper learning rule A : ( X × Y ) ∗ → H such that for every distribution D over X × Y , with probability at least 1 -δ over S ∼ D m ,

We say that H is proper probabilistically robustly PAC learnable with respect to /lscript G ,µ if ∀ /epsilon1 , δ ∈ (0 , 1) , n ( /epsilon1 , δ ; H , /lscript G ,µ ) is finite.

/negationslash

An important class of proper learning rules is ERMs which simply output the hypothesis h ∈ H that minimizes the loss /lscript G ,µ over the training sample. In this paper, we ultimately show that for a wide family of probabilistically robust loss functions, ERM is a proper learner according to Definition 1. On the other hand, in Section 3, we show that proper probabilistically robust PAC learning is not always possible for the loss function /lscript ρ G ,µ ( h, ( x, y )) .

2.3 Complexity Measures

Under the standard 0-1 risk, the Vapnik-Chervonenkis dimension (VC dimension) plays an important role in characterizing PAC learnability, and more specifically, when ERM is possible. A hypothesis class H is PAC learnable with respect to the 0-1 loss if and only if its VC dimension is finite [Vapnik and Chervonenkis, 1971].

Definition 2 (VC dimension) . A set { x 1 , ..., x n } ∈ X is shattered by H , if ∀ y 1 , ..., y n ∈ Y , ∃ h ∈ H , such that ∀ i ∈ [ n ] , h ( x i ) = y i . The VC dimension of H , denoted VC ( H ) , is defined as the largest natural number n ∈ such that there exists a set { x 1 , ..., x n } ∈ X that is shattered by H .

One sufficient condition for proper learning, based on Vapnik's 'General Learning" [Vapnik, 2006], is the finiteness of the VC dimension of a binary loss class

where /lscript ( h, ( x, y )) is some loss function mapping to { 0 , 1 } . In particular, if the VC dimension of the loss class L H is finite, then H is PAC learnable with respect to /lscript using ERM with sample complexity that scales linearly with VC ( L H ) . In this sense, if one can upper bound VC ( L H ) in terms of VC ( H ) , then finite VC dimension is sufficient for proper learnability. Unfortunately, for adversarially robust learning, Montasser et al. [2019] show that there can be an arbitrary gap between the VC dimension of the adversarially robust loss class L H G := { ( x, y ) ↦→ /lscript G ( h, ( x, y )) : h ∈ H} and the VC dimension of H . Likewise, in Section 3, we show that for the ρ -probabilistically robust loss /lscript ρ G ,µ , there can also be an arbitrarily large gap between the VC dimension of the loss class and the VC dimension of the hypothesis class.

As many of the loss functions we consider will actually map to values in , the VC dimension of the loss class will not be well-defined. Instead, we can capture the complexity of the loss class via the empirical Rademacher complexity.

Definition 3 (Empirical Rademacher Complexity of Loss Class) . Let /lscript be a loss function, S = { ( x 1 , y 1 ) , ..., ( x n , y n ) } ∈ ( X × Y ) ∗ be any sequence of examples, and L H = { ( x, y ) ↦→ /lscript ( h, ( x, y )) : h ∈ H} be a loss class. The empirical Rademacher complexity of L H is defined as

where σ 1 , ..., σ m are independent Rademacher random variables.

A standard result relates the empirical Rademacher complexity to the generalization error of hypotheses in H with respect to a real-valued bounded loss function /lscript ( h, ( x, y )) [Bartlett and Mendelson, 2002].

Theorem 2.1 (Rademacher-based Uniform Convergence) . Let D be a distribution over X × Y and /lscript ( h, ( x, y )) ≤ c be a bounded loss function. With probability at least 1 -δ over the sample S ∼ D m , for all h ∈ H simultaneously, where ˆ S [ /lscript ( h ( x ) , y )] = 1 | S | ∑ ( x,y ) ∈ S /lscript ( h ( x ) , y ) is the empirical average of the loss over S .

3 Not All Robust Loss Relaxations Enable Proper Learning

Recall the ρ -probabilistically robust loss

/negationslash

and its corresponding risk R ρ G ,µ ( h ; D ) := E ( x,y ) ∼ D [ /lscript ρ G ,µ ( h, ( x, y )) ] . At a high-level, proper learning under the /lscript ρ G ,µ requires finding a hypothesis h ∈ H that is robust to at least a 1 -ρ fraction of the perturbations in G for each example in the support of the data distribution D .

Which hypothesis classes are properly learnable with respect to /lscript ρ G ,µ according to Definition 1? In Appendix B.1, we show that if G is finite, then finite VC dimension is sufficient for proper learning with respect to /lscript ρ G ,µ . On the other hand, in this section, we show that if G is allowed to be arbitrary, VC dimension is not sufficient for proper learning with respect to /lscript ρ G ,µ , let alone learning via ERM.

Theorem 3.1. There exists ( G , µ ) such that for every ρ ∈ [0 , 1) , there exists a hypothesis class H ⊆ Y X with VC ( H ) ≤ 1 such that H is not properly probabilistically robust PAC learnable with respect to /lscript ρ G ,µ .

To prove Theorem 3.1, we fix X = d , G = { g δ : δ ∈ d , || δ || p ≤ γ } such that g δ ( x ) = x + δ for all x ∈ X for some γ > 0 , and µ to be the uniform measure over G . In other words, we are picking our perturbation sets to be /lscript p balls of radius γ and our perturbation measures to be uniform over each perturbation set. Note that by construction of G , a uniform measure µ over G also induces a uniform measure µ x over G ( x ) := { g δ ( x ) : g δ ∈ G} ⊂ d . We start by showing that for every ρ ∈ [0 , 1) , there can be an arbitrary gap between the VC dimension of H and the loss class L H , ρ G ,µ := { ( x, y ) ↦→ /lscript ρ G ,µ ( h, ( x, y )) : h ∈ H} .

Lemma 3.2. For every ρ ∈ [0 , 1) and m ∈ , there exists a hypothesis class H ⊆ Y X such that VC ( H ) ≤ 1 but VC ( L H , ρ G ,µ ) ≥ m .

The proof of Lemma 3.2 is found in Appendix B.2. We highlight two key differences between Lemma 3.2 and its analog, Lemma 2, in Montasser et al. [2019]. First, we need to provide both a perturbation set and a perturbation measure. The interplay between these two objects is not present in Montasser et al. [2019] and, apriori, it is not clear that these would indeed be /lscript p balls and the uniform measure. Second, in order for a hypothesis to be probabilistically non-robust there needs to exist a large enough region of perturbations over which it makes mistakes. This is in contrast to Montasser et al. [2019], where a hypothesis is adversarially non-robust as long as there exists one non-robust perturbation. Constructing a hypothesis class that achieves all possible probabilistically robust loss behaviors while also having low VC dimension is non-trivial - we need hypotheses to be expressive enough to have large regions of non-robustness while not being too expressive such that VC dimension increases.

Next, we show that the hypothesis class construction in Lemma 3.2 can be used to show the existence of a hypothesis class that cannot be learned properly. Lemma 3.3 is similar to Lemma 3 in Montasser et al. [2019] and is proved in Appendix B.3.

Lemma 3.3. For every ρ ∈ [0 , 1) and m ∈ there exists H ⊆ Y X with VC ( H ) ≤ 1 such that for any proper learner A : ( X × Y ) ∗ → H : (1) there is a distribution D over X × Y and a hypothesis h ∗ ∈ H where R ρ G ,µ ( h ∗ ; D ) = 0 and (2) with probability at least 1 / 7 over S ∼ D m , R ρ G ,µ ( A ( S ); D ) > 1 / 8 .

Finally, the proof of Theorem 3.1 uses Lemma 3.3 and follows a similar idea as its analog in Montasser et al. [2019] (Theorem 1). However, since our hypothesis class construction in Lemma 3.2 is different, some subtle modifications need to be made. We include a complete proof in Appendix B.4.

4 Proper Learnability Under Relaxed Losses

Despite the fact that VC classes are not ρ -probabilistically robust learnable using proper learning rules, our framework still enables us to capture a wide range of relaxations to the adversarially robust loss for which proper learning is possible.

In particular, consider robust loss relaxations of the form /lscript G ,µ ( h, ( x, y )) = /lscript ( y g ∼ µ [ h ( g ( x ))]) where /lscript ( t ) : → is a L -Lipschitz function. This class of loss functions is general, capturing many natural robust loss relaxations like the hinge loss 1 -y g ∼ µ [ h ( g ( x ))] , squared loss ( y -

/negationslash

g ∼ µ [ h ( g ( x ))]) 2 = (1 -y g ∼ µ [ h ( g ( x ))]) 2 , and exponential loss e -y g ∼ µ [ h ( g ( x ))] . Furthermore, the class of Lipschitz functions /lscript : → on the margin y g ∼ µ [ h ( g ( x ))] enables us to capture levels of robustness between the average- and worst-case. For example, taking /lscript ( t ) = 1 -t 2 results in the loss /lscript G ,µ ( h, ( x, y )) = /lscript ( y g ∼ µ [ h ( g ( x ))]) = g ∼ µ [ h ( g ( x )) = y ] , corresponding to averagecase robustness, or data augmentation . On the other hand, taking /lscript ( t ) = min( 1 -t 2 ρ , 1) for some ρ ∈ (0 , 1) , results in the loss

/negationslash which corresponds to a notion of robustness that becomes stricter as ρ approaches 0 . We note that some of the losses in our family were studied by Rice et al. [2021]. However, their focus was on evaluating robustness, while ours is about (proper) learnability.

Lemma 4.1 shows that for hypothesis classes H with finite VC dimension, for any ( G , µ ) , all L -Lipschitz loss functions /lscript G ,µ ( h, ( x, y )) enjoy the uniform convergence property.

Lemma 4.1 (Uniform Convergence of Lipschitz Loss) . Let H be a hypothesis class with finite VC dimension, ( G , µ ) be a perturbation set and measure, and /lscript G ,µ ( h, ( x, y )) = /lscript ( y g ∼ µ [ h ( g ( x ))]) such that /lscript : → is a L -Lipschitz function. With probability at least 1 -δ over a sample S ∼ D n of size n = O ( VC ( H ) L 2 ln( L /epsilon1 )+ln( 1 δ ) /epsilon1 2 ) , for all h ∈ H simultaneously,

Proof. Let VC ( H ) = d and S = { ( x 1 , y 1 ) , ..., ( x m , y m ) } be a set of examples drawn i.i.d from D . Define the loss class L H G ,µ = { ( x, y ) ↦→ /lscript G ,µ ( h, ( x, y )) : h ∈ H} . Observe that we can reparameterize L H G ,µ as the composition of a L -Lipschitz function /lscript ( x ) and the function class F H G ,µ = { ( x, y ) ↦→ y g ∼ µ [ h ( g ( x ))] : h ∈ H} . By Proposition 2.1, to show the uniform convergence property of /lscript G ,µ ( h, ( x, y )) , it suffices to upper bound ˆ R m ( L H G ,µ ) = ˆ R m ( /lscript ◦ F H G ,µ ) , the empirical Rademacher complexity of the loss class. Since /lscript is L -Lipschitz, by Ledoux-Talagrand's contraction principle [Ledoux and Talagrand, 1991], it follows that ˆ R m ( L H G ,µ ) = ˆ R m ( /lscript ◦ F H G ,µ ) ≤ L · ˆ R m ( F H G ,µ ) . Thus, it actually suffices to upperbound ˆ R m ( F H G ,µ ) instead. Starting with the definition of the empirical Rademacher complexity:

where the last inequality follows from Jensen's inequality and Fubini's Theorem. Note that the quantity 1 m σ ∼ {± 1 } m [sup h ∈ H ∑ m i =1 σ i h ( g ( x i ))] is the empirical Rademacher complexity of the hypothesis class H over the sample { g ( x 1 ) , ..., g ( x m ) } first sampling from the marginal data distribution, x , and then applying the transformation

for all h ∈ H simultaneously.

Uniform convergence of Lipschitz-losses immediately implies proper learning via ERM.

Theorem 4.2. Let /lscript G ,µ ( h, ( x, y )) = /lscript ( y g ∼ µ [ h ( g ( x ))]) such that /lscript : → is a L -Lipschitz function. For every hypothesis class H , perturbation set and measure ( G , µ ) , and ( /epsilon1 , δ ) ∈ (0 , 1) 2 , the proper learning rule A ( S ) = arg min h ∈ H ˆ S [ /lscript G ,µ ( h, ( x, y ))] , for any distribution D over X × Y , achieves, with probability at least 1 -δ over a sample S ∼ D n of size n ≥ O ( VC ( H ) L 2 ln( L /epsilon1 )+ln( 1 δ ) /epsilon1 2 ) , the guarantee

At a high-level, Theorem 4.2 shows finite VC dimension is sufficient for achieving robustness between the average- and worst-case using ERM. In fact, the next theorem, whose proof can be found in Appendix C.2, shows that finite VC dimension may not even be necessary for this to be true. The proof of Theorem 4.3 involves considering the well-known infinite VC class H = { x ↦→ sign (sin( wx )) : w ∈ } and picking ( G , µ ) such that g ∼ µ [ h ( g ( x ))] is essentially constant in x for all hypothesis h ∈ H .

Theorem 4.3. Let /lscript G ,µ ( h, ( x, y )) = /lscript ( y g ∼ µ [ h ( g ( x ))]) such that /lscript : → is a L -Lipschitz function. There exists H and ( G , µ ) such that VC ( H ) = ∞ but H is still properly learnable with respect to /lscript G ,µ ( h, ( x, y )) .

Together, Theorems 4.2 and 4.3 showcase an interesting trade-off. Theorem 4.3 indicates that by carefully choosing ( G , µ ) , the complexity of H can be essentially smoothed out. On the other hand, Theorem 4.2 shows that any complexity in ( G , µ ) can be smoothed out if H has finite VC dimension. This interplay between the complexities of H and ( G , µ ) closely matches the intuition of Chapelle et al. [2000] in their work on Vicinal Risk Minimization. Note that the results in this section do not contradict that of Section 3 because /lscript ρ G ,µ ( h, ( x, y )) is a non-Lipschitz function of y g ∼ µ [ h ( g ( x ))] .

We end this section by noting that Lipschitzness of /lscript G ,µ is sufficient but, in full generality, not necessary for proper learnability. For example, the loss function that completely ignores ( G , µ ) and just computes the 0-1 loss is not Lipschitz, however, it is learnable via ERM when the VC dimension of H is finite.

5 Proper Learnability Under Relaxed Competition

The results of Section 3 show that relaxing the adversarially robust loss may not always enable proper learning, even for very natural robust loss relaxations. In this section, we show that this bottleneck can be alleviated if we also allow the learner to compete against a slightly stronger notion of robustness. Furthermore, we expand on this idea by exploring other robust learning settings where allowing the learner to compete against a stronger notion of robustness enables proper learnability. We denote this type of modification to the standard adversarially and probabilistically robust leaning settings as robust learning under relaxed competition . Prior works on Tolerantly Robust PAC Learning [Ashtiani et al., 2022, Bhattacharjee et al., 2022] mentioned in the introduction fit under this umbrella.

Our main tool in this section is Lemma 5.1, which we term as Sandwich Uniform Convergence (SUC). Roughly speaking, SUC provides a sufficient condition under which ERM outputs a predictor that generalizes well with respect to a stricter notion of loss. A special case of SUC has implicitly appeared in margin theory (e.g., see Mohri et al. [2018, Section 5.4]), where one evaluates the 0 -1 risk of the output hypothesis against the optimal margin 0 -1 risk.

Lemma 5.1 (Sandwich Uniform Convergence) . Let /lscript 1 ( h, ( x, y )) and /lscript 2 ( h, ( x, y )) be bounded, nonnegative loss functions such that for all h ∈ H and ( x, y ) ∈ X × Y , we have /lscript 1 ( h, ( x, y )) ≤ /lscript 2 ( h, ( x, y )) ≤ 1 . If there exists a loss function ˜ /lscript ( h, ( x, y )) such that /lscript 1 ( h, ( x, y )) ≤ ˜ /lscript ( h, ( x, y )) ≤ /lscript 2 ( h, ( x, y )) and ˜ /lscript ( h, ( x, y )) enjoys the uniform convergence property with sample complexity n ( /epsilon1 , δ ) , then the learning rule A ( S ) = inf h ∈ H ˆ S [ /lscript 2 ( h, ( x, y ))] achieves, with probability 1 -δ over a sample S ∼ D m of size m ≥ n ( /epsilon1 / 2 , δ / 2) + O ( ln( 1 δ ) /epsilon1 2 ) , the guarantee

Proof. Let A ( S ) = inf h ∈ H E S [ /lscript 2 ( h, ( x, y ))] . By uniform convergence of ˜ /lscript ( h, ( x, y )) , we have that for sample size m = n ( /epsilon1 2 , δ 2 ) , with probability at least 1 -δ 2 , over a sample S ∼ D m , for every hypothesis h ∈ H simultaneously, where h ∗ = inf h ∈ H E D [ /lscript 2 ( h, ( x, y ))] . It now remains to upper bound ˆ E S [ /lscript 2 ( h ∗ , ( x, y ))] with high probability. However, a standard Hoeffding bound tells us that with probability 1 -δ 2 over a sample S of size O ( ln( 1 δ ) /epsilon1 2 ) , ˆ E S [ /lscript 2 ( h ∗ , ( x, y ))] ≤ D [ /lscript 2 ( h ∗ , ( x, y ))] + /epsilon1 2 . Thus, by union bound, we get that with probability at least 1 -δ , D [ /lscript 1 ( ˆ h, ( x, y )) ] ≤ D [ /lscript 2 ( h ∗ , ( x, y ))] + /epsilon1 , using a sample of size n ( /epsilon1 / 2 , δ / 2) + O ( ln( 1 δ ) /epsilon1 2 ) .

D [ ˜ /lscript ( h, ( x, y )) ] ≤ ˆ E S [ ˜ /lscript ( h, ( x, y )) ] + /epsilon1 2 . In particular, this implies that for ˆ h = A ( S ) , we have D [ ˜ /lscript ( ˆ h, ( x, y )) ] ≤ ˆ E S [ ˜ /lscript ( ˆ h, ( x, y )) ] + /epsilon1 2 . Since, /lscript 1 ( h, ( x, y )) ≤ ˜ /lscript ( h, ( x, y )) ≤ /lscript 2 ( h, ( x, y )) , we have that D [ /lscript 1 ( ˆ h, ( x, y )) ] ≤ ˆ E S [ /lscript 2 ( h ∗ , ( x, y ))] + /epsilon1 2

Lemma 5.1 only requires the existence of such a sandwiched loss function that enjoys uniform convergence-we do not actually require it to be computable. In the next two sections, we exploit this fact to give three new generalization guarantees for the empirical risk minimizer over the adversarially robust loss /lscript G ( h, ( x, y )) and ρ -probabilistically robust loss /lscript ρ G ,µ ( h, ( x, y )) , hereafter denoted by RERM ( S ; G ) := arg min h ∈ H ˆ S [ /lscript G ( h, ( x, y ))] and PRERM ( S ; ( G , µ ) , ρ ) := arg min h ∈ H ˆ S [ /lscript ρ G ,µ ( h, ( x, y )) ] respectively.

5.1 ( ρ , ρ ∗ ) -Probabilistically Robust PAC Learning

In light of the hardness result of Section 3, we slightly tweak the learning setup in Definition 1 by allowing A to compete against the hypothesis minimizing the probabilistic robust risk at a level ρ ∗ < ρ . Under this further relaxation, we show that proper learning becomes possible, and that too, via PRERM. In particular, Theorem 5.2 shows that while VC classes are not properly ρ -probabilistically robust PAC learnable, they are properly ( ρ , ρ ∗ ) -probabilistically robust PAC learnable.

Theorem 5.2 (Proper ( ρ , ρ ∗ ) -Probabilistically Robust PAC Learner) . Let 0 ≤ ρ ∗ < ρ . Then, for every hypothesis class H , perturbation set and measure ( G , µ ) , and ( /epsilon1 , δ ) ∈ (0 , 1) 2 , the proper learning rule A ( S ) = PRERM ( S ; ( G , µ ) , ρ ∗ ) , for any distribution D over X × Y , achieves, with probability at least 1 -δ over a sample S ∼ D n of size n ≥ O ( VC ( H ) ( ρ -ρ ∗ ) 2 ln( 1 ( ρ -ρ ∗ ) /epsilon1 )+ln( 1 δ ) /epsilon1 2 ) , the guarantee

/negationslash

In contrast to Section 3, where proper learning is not always possible, Theorem 5.2 shows that if we compare our learner to the best hypothesis for a slightly stronger level of probabilistic robustness, then not only is proper learning possible for VC classes, but it is possible via ERM. Our main technique to prove Theorem 5.2 is to consider a different probabilistically robust loss function that is (1) a Lipschitz function of y g ∼ µ [ h ( g ( x )) = y ] and (2) can be sandwiched in between /lscript ρ ∗ G ,µ and /lscript ρ G ,µ . Then, Theorem 5.2 follows from Lemma 5.1. The full proof is in Appendix D.1.

5.2 ( ρ , G ) -Probabilistically Robust PAC Learning

Can measure-independent learning guarantees be achieved if we instead compare the learner's probabilistically robust risk R ρ G ,µ to the best adversarially robust risk R G over H ? We answer this in the affirmative by using SUC. We show that if one wants to compete against the best hypothesis for the worst-case adversarially robust risk, it is sufficient to run RERM.

Theorem 5.3 (Proper ( ρ , G ) -Probabilistically Robust PAC Learner) . For every hypothesis class H , perturbation set G , and ( /epsilon1 , δ ) ∈ (0 , 1) 2 , the proper learning rule A ( S ) = RERM ( S ; G ) , for any measure µ over G and any distribution D over X × Y , achieves, with probability at least 1 -δ over a sample S ∼ D n of size n ≥ O ( VC ( H ) ρ 2 ln( 1 ρ/epsilon1 )+ln( 1 δ ) /epsilon1 2 ) , the guarantee

The proof of Theorem 5.3 can be found in Appendix D.2, which follows directly from Lemma 5.1 by a suitable choice of the sandwiched loss /lscript . We make a few remarks about the practical importance of Theorem 5.3. Theorem 5.3 implies that for any pre-specified perturbation function class G (for example /lscript p balls), running RERM is sufficient to obtain a hypothesis that is probabilistically robust with respect to any fixed measure µ over G . Moreover, the level of robustness of the predictor output by RERM, as measured by 1 -ρ , scales directly with the sample size - the more samples one has, the smaller ρ can be made. Alternatively, for a fixed sample size m , desired error /epsilon1 and confidence δ , one can use the sample complexity guarantee in Theorem 5.3 to back-solve the robustness guarantee ρ .

We highlight that the generalization bound in Theorem 5.3, is a measure-independent guarantee. This means that ρ does not quantify the level of robustness of the output hypothesis with respect to any one particular measure, but for any measure. This is desirable, as in contrast to the previous section, the ρ here more succinctly quantifies the level of robustness achieved by the output classifier. Lastly, we highlight that while the sample complexity of adversarially robust PAC learning can be exponential in the VC dimension of H [Montasser et al., 2019], this is not the case for ( ρ , G ) -probabilistically robust PAC learning, where we only get a linear dependence on VC dimension.

5.3 Tolerantly Robust PAC Learning

In Tolerantly Robust PAC Learning [Bhattacharjee et al., 2022, Ashtiani et al., 2022], the learner's adversarially robust risk under a perturbation set G is compared with the best achievable adversarially robust risk for a larger perturbation set G ′ ⊃ G . Ashtiani et al. [2022] study the setting where both G and G ′ induce /lscript p balls with radius r and (1 + γ ) r respectively. In the work of Bhattacharjee et al. [2022], G is arbitrary, but G ′ is constructed such that it induces perturbation sets that are the union of balls with radius γ that cover G . Critically, Bhattacharjee et al. [2022] show that, under certain assumptions, running RERM over a larger perturbation set G ′ is sufficient for Tolerantly Robust PAC learning. In this section, we take a slightly different approach to Tolerantly Robust PAC learning. Instead of having the learner compete against the best possible risk for a larger perturbation set, we have the learner still compete against the best possible adversarially robust risk over G , but evaluate the learner's adversarially robust risk using a smaller perturbation set G ′ ⊂ G .

For what G ′ ⊂ G is Tolerantly Robust PAC learning via RERM possible? As an immediate result of Lemma 5.1 and Vapnik's 'General Learning', finite VC dimension of the loss class L H G ′ = { ( x, y ) ↦→ /lscript G ′ ( h, ( x, y )) : h ∈ H} is sufficient. Note that finite VC dimension of L H G ′ implies that the loss function /lscript G ′ ( h, ( x, y )) enjoys the uniform convergence property with sample complexity O ( VC ( L G ′ H )+ln( 1 δ ) /epsilon1 2 ) . Thus, taking /lscript 1 ( h, ( x, y )) = ˜ /lscript ( h, ( x, y )) = /lscript G ′ ( h, ( x, y )) and /lscript 2 ( h, ( x, y )) = /lscript G ( h, ( x, y )) in Lemma 5.1, we have that if there exists a G ′ ⊂ G such that VC ( L G ′ H ) < ∞ , then with probability 1 -δ over a sample S ∼ D n of size n = O ( VC ( L H G ′ )+ln( 1 δ ) /epsilon1 2 ) ,

where A ( S ) = RERM ( S ; G ) .

Alternatively, if G ′ ⊂ G such that there exists a finite subset ˜ G ⊂ G where /lscript G ′ ( h, ( x, y )) ≤ /lscript ˜ G ( h, ( x, y )) , then Tolerantly Robust PAC learning via RERM is possible with sample complexity that scales according to O ( VC ( H ) log( | ˜ G| )+ln( 1 δ ) /epsilon1 2 ) . This result essentially comes from the fact that the VC dimension of the loss class for any finite perturbation set ˜ G incurs only a log( | ˜ G| ) blow-up from the VC dimension of H (see Lemma 1.1 in Attias et al. [2021]). Thus, finite VC dimension of

H implies finite VC dimension of the loss class L ˜ G H which implies uniform convergence of the loss /lscript ˜ G ( h, ( x, y )) , as needed for Lemma 5.1 to hold.

We now give an example where such a finite approximation of G ′ is possible. In order to do so, we will need to consider a metric space of perturbation functions ( G , d ) and define a notion of 'nice' perturbation sets, similar to'regular' hypothesis classes from Bhattacharjee et al. [2022].

Definition 4 ( r -Nice Perturbation Set) . Let H be a hypothesis class and ( G , d ) a metric space of perturbation functions. Let B r ( g ) := { g ′ ∈ G : d ( g, g ′ ) ≤ r } denote a closed ball of radius r centered around g ∈ G . We say that G ′ ⊂ G is r -Nice with respect to H , if for all x ∈ X , h ∈ H , and g ∈ G ′ , there exists a g ∗ ∈ G , such that g ∈ B r ( g ∗ ) and h ( g ( x )) = h ( g ′ ( x )) for all g ′ ∈ B r ( g ∗ ) .

Definition 4 prevents a situation where a hypothesis h ∈ H is non-robust to an isolated perturbation function g ∈ G ′ for any given labelled example ( x, y ) ∈ X × Y . If a hypothesis h is non-robust to a perturbation g ∈ G ′ , then Definition 4 asserts that there must exist a small ball of perturbation functions in G over which h is also non-robust. Next, we define the covering number.

Definition 5 (Covering Number) . Let ( M , d ) be a metric space, let K ⊂ M be a subset, and r > 0 . Let B r ( x ) = { x ′ ∈ M : d ( x, x ′ ) ≤ r } denote the ball of radius r centered around x ∈ M . A subset C ⊂ M is an r -covering of K if K ⊂ ⋃ c ∈ C B r ( c ) . The covering number of K , denoted N r ( K , d ) , is the smallest cardinality of any r -covering of K .

Finally, let G ′ 2 r = ⋃ g ∈ G ′ B 2 r ( g ) denote the union over all balls of radius 2 r with centers in G ′ . Theorem 5.4 then states that if there exists a set G ′ ⊂ G that is r -Nice with respect to H , then Tolerantly Robust PAC learning is possible via RERM with sample complexity that scales logarithmically with N r ( G ′ 2 r , d ) .

Theorem 5.4 (Tolerantly Robust PAC learning under Nice Perturbations) . Let H ⊆ Y X be a hypothesis class and ( G , d ) be a metric space of perturbation functions. Given a subset G ′ ⊂ G such that G ′ is r -Nice with respect to H , then the proper learning rule A ( S ) = RERM ( S ; G ) , for any distribution D over X × Y , achieves, with probability at least 1 -δ over a sample S ∼ D n of size n ≥ O ( VC ( H ) log( N r ( G ′ 2 r ,d ))+ln( 1 δ ) /epsilon1 2 ) , the guarantee

In Appendix D, we give a full proof and show that /lscript p balls are r -Nice perturbation sets for robustly learning halfspaces . Note that Theorem 5.4 does not require apriori knowledge of the r -Nice perturbation set G ′ , but just its existence . Therefore, Theorem 5.4 applies to the largest possible r -Nice perturbation subset of G . This is important from a practical standpoint as computing an r -Nice perturbation set might not be computationally tractable. Accordingly, while RERM may not be a proper adversarially robust PAC learner [Montasser et al., 2019], Theorem 5.4 shows that RERM is a proper tolerantly robust PAC learner.

6 Discussion

In this work, we show that there exists natural relaxations of the adversarially robust loss for which finite VC dimension is still not sufficient for proper learning. On the other hand, we identify a large set of Lipschitz robust loss relaxations for which finite VC dimension is sufficient for proper learning. In addition, we give new generalization guarantees for RERM. As future work, we are interested in understanding whether our robust loss relaxations can be used to mitigate the tradeoff between achieving adversarial robustness and maintaining high nominal performance. In addition, it is also interesting to find a combinatorial characterization of proper probabilistically robust PAC learning with respect to /lscript ρ G ,µ .

References

Hassan Ashtiani, Vinayak Pathak, and Ruth Urner. Black-box certification and learning under adversarial perturbations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 388-398. PMLR, 13-18 Jul 2020.

Hassan Ashtiani, Vinayak Pathak, and Ruth Urner. Adversarially robust learning with tolerance. arXiv preprint arXiv:2203.00849 , 2022.

Idan Attias and Steve Hanneke. Adversarially robust pac learnability of real-valued functions. 2023.

Idan Attias, Aryeh Kontorovich, and Yishay Mansour. Improved generalization bounds for adversarially robust learning. 2021.

Idan Attias, Steve Hanneke, and Yishay Mansour. A characterization of semi-supervised adversarially robust pac learnability. Advances in Neural Information Processing Systems , 35:23646-23659, 2022.

Pranjal Awasthi, Anqi Mao, Mehryar Mohri, and Yutao Zhong. H-consistency bounds for surrogate loss minimizers. In International Conference on Machine Learning , pages 1117-1174. PMLR, 2022a.

Pranjal Awasthi, Anqi Mao, Mehryar Mohri, and Yutao Zhong. Multi-class h -consistency bounds. Advances in neural information processing systems , 35:782-795, 2022b.

Pranjal Awasthi, Anqi Mao, Mehryar Mohri, and Yutao Zhong. Theoretically grounded loss functions and algorithms for adversarial robustness. In International Conference on Artificial Intelligence and Statistics , pages 10077-10094. PMLR, 2023.

Han Bao, Clay Scott, and Masashi Sugiyama. Calibrated surrogate losses for adversarially robust classification. In Conference on Learning Theory , pages 408-451. PMLR, 2020.

Peter Bartlett. Lecture notes in theoretical statistics, 2013.

Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463-482, 2002.

Robi Bhattacharjee, Max Hopkins, Akash Kumar, Hantao Yu, and Kamalika Chaudhuri. Robust empirical risk minimization with tolerance. arXiv preprint arXiv:2210.00635 , 2022.

Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. Advances in neural information processing systems , 13, 2000.

Daniel Cullina, Arjun Nitin Bhagoji, and Prateek Mittal. Pac-learning in the presence of adversaries. Advances in Neural Information Processing Systems , 31, 2018.

Edgar Dobriban, Hamed Hassani, David Hong, and Alexander Robey. Provable tradeoffs in adversarially robust classification. arXiv preprint arXiv:2006.05161 , 2020.

Cassidy Laidlaw and Soheil Feizi. Functional adversarial attacks. Advances in neural information processing systems , 32, 2019.

Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: isoperimetry and processes , volume 23. Springer Science & Business Media, 1991.

Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. Tilted empirical risk minimization. arXiv preprint arXiv:2007.01162 , 2020.

Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. On tilted losses in machine learning: Theory and applications. arXiv preprint arXiv:2109.06141 , 2021.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017.

A Mao, M Mohri, and Y Zhong. Cross-entropy loss functions: Theoretical analysis and applications. arxiv. arXiv preprint arXiv:2304.07288 , 2023.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Omar Montasser, Steve Hanneke, and Nathan Srebro. Vc classes are adversarially robustly learnable, but only improperly. In Conference on Learning Theory , pages 2512-2530. PMLR, 2019.

Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial examples. arXiv preprint arXiv:1801.09344 , 2018.

Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C Duchi, and Percy Liang. Adversarial training can hurt generalization. arXiv preprint arXiv:1906.06032 , 2019.

Leslie Rice, Anna Bair, Huan Zhang, and J Zico Kolter. Robustness between the worst and average case. Advances in Neural Information Processing Systems , 34:27840-27851, 2021.

Alexander Robey, Luiz FO Chamon, George J Pappas, and Hamed Hassani. Probabilistically robust learning: Balancing average-and worst-case performance. arXiv preprint arXiv:2202.01136 , 2022.

Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adversarially robust generalization requires more data. Advances in neural information processing systems , 31, 2018.

Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin-Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy?-a comprehensive study on the robustness of 18 deep image classification models. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 631-648, 2018.

Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.12152 , 2018.

Vladimir Vapnik. Estimation of dependences based on empirical data . Springer Science & Business Media, 2006.

Vladimir Naumovich Vapnik and Aleksei Yakovlevich Chervonenkis. On uniform convergence of the frequencies of events to their probabilities. Teoriya Veroyatnostei i ee Primeneniya , 16(2):264-279, 1971.

Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International conference on machine learning , pages 5286-5295. PMLR, 2018.

Yao-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R Salakhutdinov, and Kamalika Chaudhuri. A closer look at accuracy vs. robustness. Advances in neural information processing systems , 33:8588-8601, 2020.

Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning , pages 7472-7482. PMLR, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper initiates the study of a new kind of PAC learning: probabilistically robust PAC learning.  The authors show that the finiteness of the VC dimension of the function class is not sufficient to obtain a proper learning rule in this new PAC learning setup.  However, they show that for Lipschitz losses that interpolate between the average and worst case, proper learning is possible given finite VC dimension.  They also consider the settings of adversarially robust PAC learning and tolerant PAC learning; ultimately, this results in several extensions to previous works on these topics.\n\nSTRENGTHS:\n**New problem setting.**  This is a new problem.  The authors consider several recent results in the robustness literature concerning robustness between the average and worst case, and they form a new definition of probabilistically robust PAC learning.  I imagine that this is a direction that other may be interested in, and therefore this constitutes an interesting contribution.\n\n**Non-proper learnability.**  Perhaps the most interesting/surprising result here is that the finiteness of the VC dimension of the function class is not sufficient for proper $\\rho$-probabilistic robust PAC learning.  This implies that both probabilistic and adversarial robustness do not easily admit proper learning rules.  This is somewhat surprising, given previous empirical results that find that probabilistic robustness does not come at the cost of degraded nominal performance with respect to an empirical risk minimizer.  Building on this, it is also interesting that there do exist interpolation schemes that do admit proper learning rules given finite VC dimension.  In this way, this paper is a first step toward characterizing a hierarchy of interpolating losses based on which ones admit proper learning rules.  It would be interesting to know whether other interpolation methods, e.g., those in [Rice et al., 2021] and [Li et al., 2020] also engender proper learning rules.\n\n**Technical rigor.**  The paper seems technically sound to me.  The proofs in the appendix are well-structured, and the array of tools used in the appendix may be of broader interest to the community.\n\nWEAKNESSES:\n**Unevenness of the presentation.**  There's a certain unevenness about the presentation of the main results.  In general, the trend is that as one gets further into the paper, the results get harder to parse.  This doesn't seem to be a function of the complexity of the results; rather, it seems as if less space was dedicated to fully explaining the results that appear later in the paper.  \n\nFor instance, the results in Section 3 are outlined in detail.  Theorem 3 tells us that there exists hypothesis classes for which $\\rho$-probabilistically robust PAC learning is not possible, and the proof is sketched almost in full.  However, by the time we reach Section 5, the results are stated with less explanation.  The text becomes quite difficult to read because nearly every equation on pages 8-9 is inline.  Definitions and theorems seem to be crammed in, and the paper ends without discussing the implications of the final theorem.  I think that readibility would be greatly improved if the authors moved the proofs from Section 3 to the appendix, and (i) expanded more on the results later in the paper, (ii) broke up the texts by putting long equations on their own lines (i.e., not in-line), and (iii) adding a discussion section where the implications of the results are discussed.  In its current form, the paper ends rather abruptly, and I think point (iii) would help to ameliorate this.  \n\n**Writing and notation.**  I think that the presentation could be improved in several ways.  Here are some points that occurred to me while reading the paper:\n\n* It would also be helpful if the authors could use equation numbers.\n* It's confusing that the authors introduce VC dimension and Rademacher complexity, but a definition arguably more central to this paper -- the definition of a *proper* learning rule -- was omitted.\n* ERM seems to be used to denote \"empirical risk minimization\" (throughout) and \"empirical risk minimizer\" (line 95).  I think it'd be worth picking one.\n* What is $(\\mathcal{X}\\times\\mathcal{Y})^\\star$, i.e., what does the *\\star* denote?\n* Why is there a change in notation from $\\mathcal{L}^{\\mathcal{H}}$ to to $\\mathcal{F}$ in Definition 2?\n\n**Section 5.3.**  When reading, I wasn't sure how Section 5.3 fit in with the rest of the paper.  Whereas the other sections of the paper focus on proper learnability for probabilistic robustness and generalizations which interpolate between average and worst-case robustness, Section 5.3 seems to address questions that are somewhat different in spirit.  I suppose one could argue that in the tolerance setting, reducing $\\mathcal{G}$ to a singleton set containing the identify function would show that this paradigm can interpolate between the standard PAC model and robust variants, but this connection feels tenuous.  Perhaps the authors could elaborate more on why this section fits in with this paper.\n\nQUESTIONS:\nIs it fair to say that the fact that the probabilistic robustness loss function is non-Lipschitz is the reason behind Theorem 3.1.  In other words, is Lipschitzness a necessary condition?  My understand from Section 4 is that it is sufficient, but it may not be necessary.\n\n**Overall assessment.**  Overall, I do not see a strong reason for this paper not to be accepted.  It studies a new problem and the insights are novel and interesting.  There are some drawbacks regarding the presentation, but one imagines that these can be easily ironed out.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-13"}

comments: SUMMARY:
This paper investigates the relaxations of the worst-case robust loss to make VC classes properly PAC learnable. Firstly, this paper shows that an exsiting and natural relaxation does not work. Then, the paper gives a family of robust loss relaxations that interpolate between average- and worst-case robustness. Finally, the paper studies the generalization guarantees for the adversarially robust empirical risk minimizer.

STRENGTHS:
- This paper shows that an exsiting and natural relaxation does not work.
- This paper gives a family of robust loss relaxations that interpolate between average- and worst-case robustness and shows that they make the VC classes properly learnable. The results are interesting.

WEAKNESSES:
- Lack of descriptions about the high-level intuitions (please refer to the questions part).
- Some minor issues. The label space is defined as $\mathcal{Y} = \\{ -1, 1 \\}$, however, in the proof of Lemma 3.2, the paper uses $\\{ 0,1 \\}$. In Lemma 4.1, it seems that $\ell$ needs to be bounded but the paper ignores it.

QUESTIONS:
- Would you please show some ideas about the proof of Lemma 3.2? When considering this problem, is it the case that you first consider giving an upper bound of the VC dimension of $\mathcal{L}$ in terms of the VC dimension of $\mathcal{H}$ or the case that you directly try to construct the counterexample? Would you please show some high-level thoughts about adapting the proof of Omar to the case in this paper?
- Would you please provide some high-level insights about the construction of the counterexample in Theorem 4.3?

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-22

comments: SUMMARY:
This paper studies the proper robust learnability under relaxation of the (usual) worst-case/all powerful adversary assumption. 

- The authors first show that finite VC dimension is not sufficient to enable proper learnability under the relaxation proposed by Robey et al. (2022).
- For another generalization of worst-case relaxations, the authors show that finite VC dimension enables proper robust learnability 
- The authors study the "relaxed competition" setting where the hypothesis is compared to the optimal hypothesis under a slightly stronger notion of robustness, proper learning guarantees are possible

STRENGTHS:
- The paper is well-written, clear and easy to follow
- I believe the topic and results are of interest to the learning theory community, 
- Relaxing the worst-case analysis is well-motivated

WEAKNESSES:
1. It seems perhaps a considerable number of proofs rely on standard techniques
2. Is it a limitation / too big of a relaxation to have the adversary pick a perturbation independently of the unperturbed point $x$? (l.70-71) Unless $\mu$ can be defined for each $x$. Either way, it would be worth discussing and clarifying this point in the main body (unless I have missed this somewhere).  

Overall I think even if the potential limitations pointed out above are right, the paper still offers a good contribution.

QUESTIONS:
Could you address point 2 above?

LIMITATIONS:
If point 2 in "Weaknesses" is correct, it would be worth including as a limitation of the work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the setting of robust PAC learning to test time attacks, using a relaxed notion of robustness on average instead of robustness to the worst-case attack.  \n\nThe contributions are as follows. \n\n-Negative result: even when using the relaxed notion of robustness, improper learning is impossible. This is a stronger result from the example in the worst-case setting [Montasser et al. 2019]. Moreover, this is achieved by a natural example of $\\ell_p$ balls and the uniform measure. \nThe intuition is that the non-Lipschitzness of the 0-1 loss enforces to use improper learning. \n\n-Positive results: \n1. When considering Lipschitz loss functions, uniform convergence hold, and as a result, ERM is sufficient for learnability. \n2. Instead of relaxing the robust loss, it is possible to relax the benchmark we compare to, i.e. the best function in the class but with a smaller parameter in the probabilistic loss. This is similar to the setup of Tolerant Robust PAC Learning.\n\nSTRENGTHS:\nThis paper provides nice contributions to the literature on robust learning, by finding natural relaxations on the robust model that allows learning similar to non-robust learning.\n\nWEAKNESSES:\nSee Questions. \nThe writing can be improved. This paper has many good ideas, but sometimes it is hard to follow them.\nAlso, many relevant references from theory on robust learning are missing that should be included as related work.\n\nQUESTIONS:\nThe description of the model might be confusing. I will explain my point of view. \nIn the standard setting, the set of possible perturbations is fixed and known to the learner. It's not chosen by an adversary, it's just the possible attacks the learner is aiming to protect from at test time. An adversary would just pick the set of all possible perturbations. \n\nIn this model, is the set G and measure $\\mu$ being chosen at training time and known to the learner? If so, that makes sense to me and should be clear in the model.\nWhat's a reasonable choice measure? is it representing the importance of each $g$? I think that some motivation is missing.\n\nThe connection between the average case and worst case model is through using the  $\\rho$-probabilistically robust loss, maybe it should be mentioned before section 3. This is a very important explanation for defining the model of robustness on average!\n\nSome definitions are used throughout the paper. It could improve the readability if those be numbered and referred to when used. \nFor example, the risk under the probabilistic robust loss (line 116) is used in section 5. It takes some time to find the definition. \n\nMany references are missing. For example, \nH-consistency bounds for surrogate loss minimizers (ICML 2022),\nMulti-class H-consistency bounds (NeurIPS 2022),\nTheoretically grounded loss functions and algorithms for adversarial robustness (AISTATS 2023),\nCross-Entropy Loss Functions: Theoretical Analysis and Applications (ICML 2023),\nA Characterization of Semi-Supervised Adversarially Robust PAC Learnability (NeurIPS 2022),\nAdversarially Robust PAC Learnability of Real-Valued Functions (ICML 2023),\nOn the hardness of robust classification (JMLR)\n...and many more!\n\nLIMITATIONS:\nThere are no limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}