Title: Characterization of Overfitting in Robust Multiclass Classification
Venue: NeurIPS 2023
Accepted: yes
Keywords: Learning Theory
Abstract: This paper considers the following question: Given the number of classes m, the number of robust accuracy queries k, and the number of test examples in the dataset n, how much can adaptive algorithms robustly overfit the test dataset? We solve this problem by equivalently giving near-matching upper and lower bounds of the robust overfitting bias in multiclass classification problems.

Characterization of Overfitting in Robust Multiclass Classification

Abstract

This paper considers the following question: Given the number of classes m, the number of robust accuracy queries k, and the number of test examples in the dataset n, how much can adaptive algorithms robustly overfit the test dataset? We solve this problem by equivalently giving near-matching upper and lower bounds of the robust overfitting bias in multiclass classification problems.

1 Introduction

Learning models that are robust to adversarial perturbations has garnered significant research attention in recent years. However, despite this progress, a pervasive issue continues to plague these models, namely robust overfitting [1]. A common approach to overcome overfitting is to divide the dataset into a training set, and a holdout (or test) set. Nonetheless, modern machine learning is adaptive in its nature. Prior information about a model's performance on the test set inevitably influences future modeling choices in extensive experiments and competitions. Recent studies have shown that excessive reuse of the holdout dataset can also leads to overfitting in non-robust setting [2, 3], and a body of subsequent work has quantitatively explored this phenomenon in the framework of perfect test label reconstruction [4, 5, 6]. Accordingly, in this paper we attempt to address the following questions: Can adaptive behavior result in overfitting in an adversarial setting? If so, by how much can adaptive algorithms robustly overfit the test dataset?

To solve these questions, we generalize the framework of perfect reconstruction to the adversarial setting, and analyze the average case performance that can be achieved by an adaptive algorithm, denoted as h U ( k, n, m ) , where k, n, m represents the number of robust accuracy queries, test samples and classes, respectively. This term equivalently measures the maximum level of robust overfitting in a multiclass classification problem. In this paper, we derive both upper and lower bounds of h U ( k, n, m ) , and demonstrate that our upper bounds and lower bounds are matching within logarithmic factors when n and the distribution of test dataset features D X are fixed.

1.1 Related works

Perfect test label reconstruction. The question of perfect test label reconstruction in non-robust setting dates back to decades ago [7, 8, 9]. The developments on studying biasing results due to adaptive reuse of the test data start with the work of [4, 10], which broadly fall in the field of adaptive data analysis [11, 12]. [5] first pose the problem of characterizing the overfitting bias as a function of k, n, m , but they fail to give upper and lower bounds on the same order of m, which is left as an open

question [13]. [6] close this question and determine the amount of overfitting possible in multiclass classification. The theory on quantitative overfitting analysis in adversarial setting remains blank.

Adversarial robustness. It has been shown that deep neural networks are fragile to imperceptible distortions in the input space [14]. Perhaps the most common method to improve adversarial robustness is adversarial training [15, 16]. Theoretically, [17, 18, 19, 20] study the PAC learnability of adversarial robust learning problem, and [21, 22] study adversarial robustness under self-supervised learning. [23, 24] investigates the adversarial robustness from the perspective of ordinary differential equations. Besides, [25] analyze the trade-off between robustness and fairness, [26] study the worst-class adversarial robustness in adversarial training.

2 Summary of our results

In the remainder of this article, R , N and R d represent the sets of real numbers, natural numbers and d -dimensional vectors over R , respectively. We denote the set { 1 , . . . , n } (for n ∈ N ) by [ n ] . If A and B are sets, we use B A to denote the collection of all mappings from A to B and 2 A to denote the power set of A, that is the collection of all subsets of A. We denote the indicator function by ✶ { event } , that is 1 if an event happens and 0 otherwise. If D is a distribution, we use supp( D ) to denote the support of D , which is defined by the closure of the set of possible values of a random variable having D as its distribution. Besides, ∥ · ∥ p represents the ℓ p norm and d p ( · , · ) represents the distance function induced by ℓ p norm. Finally, we use big tilde notations ˜ O, ˜ Ω and ˜ Θ as variants of big O notations that ignores logarithmic factors.

2.1 Problem formulation

Let X = R d be the instance space and Y = { 1 , . . . , m } be the label space. Let S = { ( x i , c i ) } n i =1 denote the test set, whose features, denoted as X S = { x 1 , . . . , x n } , are independent and identically distributed (i.i.d.) according to some distribution D X on X 2 . For simplification, we use ¯ c = ( c 1 , . . . , c n ) to describe the vector of test set labels. Let f : X → Y be a function, its robust accuracy on the test set with respect to (w.r.t.) a small perturbation U : X → 2 X is defined by

The perturbation U ( x ) is required to be nonempty, so some choice of x ′ is always available. This paper focuses the case when the perturbation is the p -norm ball with a small radius r , i.e. U ( x ) = { z ∈ X : ∥ z -x ∥ p ≤ r } for some p ≥ 1 . r = 0 gives the identity perturbation : I ( x ) ≜ { x } . Note that in this case, the definition of robust accuracy is reduced to standard (non-robust) accuracy.

In this work, we mainly study the robust overfitting attack algorithms, which do not have access to the test set S. Instead, they have query access to robust accuracy of the model on S, that is, for any classifier f, the algorithm is able to obtain the value Acc U ( f ; S ) . We refer to this access as a query . A k-query algorithm A makes k queries f 1 , . . . , f k on S, and based on the values of X S and Acc U ( f 1 ; S ) , . . . , Acc U ( f k ; S ) , A outputs a classifier ˆ f = A ( S ) . We say a k -query algorithm A is based on hypothesis class H ⊂ Y X if f 1 , . . . , f k ∈ H . The performance of the algorithm on S is measured by

where the expectation is over the algorithm's randomization. It is also of interest to ask the performance of an algorithm when ¯ c are drawn according to some distribution D Y over [ m ] n . Let D = D X ×D Y , for an algorithm A , define its performance w.r.t. D by

And we evaluate the algorithms under the assumption that they do not have any prior knowledge about the test labels. That is, the prior distribution of test labels is uniform over all possible labeling:

where µ n m denotes the uniform distribution over [ m ] n . Since the best robust accuracy is 1 /m without making any queries ( k = 0 ), h U ( A ; S ) -1 /m measures how much A robustly overfits S . The goal of this paper is to find the largest robust overfitting possible for a multiclass classification problem. So we define the performance that is achievable by an algorithm after making k queries on any S by

and we are interested in the value of h U ( k, n, m ) -1 /m, or h U ( k, n, m ) equivalently. In the rest of this paper, we aim at deriving bounds of h U ( k, n, m ) for given k, n and m.

2.2 Our main results

Our bounds on h U ( k, n, m ) have two different regimes, which can be summarized as the following theorem.

Theorem 1 (Informal) . let D X be the distribution of test sample features. For k = ˜ O ( n/m ) ,

For k = ˜ Ω( n/m ) ,

where Φ D X ( n ) ≤ 1 and is monotonically decreasing w.r.t. n.

When U = I , our upper bounds match the best known upper bounds [5], while our lower bounds differ from the known optimal lower bounds [6] by a factor Φ D X ( n ) . Since overfitting in the context of robust learning has some requirements on the test samples (e.g. the well-separated property for different classes [27]), we may not able to ensure a non-trivial lower bound on h U ( k, m, n ) for some D X . Intuitively, Φ D X ( n ) measures how easily to sample n 'good' (for robust overfitting) test data features from D X . The specific form of Φ D X ( n ) is presented in Section 3.2. Note that for a fixed size of S whose features are i.i.d. according to a fixed distribution D X , the upper and lower bounds are matching up to a logarithmic factor, that is,

and

for any fixed n and D X .

2.3 Overview of our techniques

Next, we give a brief overview of proof techniques used to obtain the main results. We first note that throughout this paper we use the notion of corrupted hypothesis [28], which transforms the formulation of robust accuracy to a non-robust one thus greatly simplifying the proofs. The definition of corrupted hypothesis is presented in the beginning of Section 3.

We establish the upper bounds via minimum description length argument, following closely a proof of an analogous result by [5] for non-robust setting. Note that their bounds can be viewed as trivial upper bounds of h U ( k, n, m ) since non-robust accuracy always upper bounds robust accuracy. We tighten the bounds by considering the query class of an algorithm. The details are presented in Theorem 2.

To obtain the lower bounds, we propose computationally efficient algorithms for two regions of k respectively. The algorithms are modified from [6], who study the worst case overfitting bias in a non-robust setting. To take the perturbation of features into account, we extend their queries to the whole feature space X by assigning each label of x ∈ X to be the 'closest' label in X S in the sense of p -norm. The theoretical guarantees are given in Theorems 3 and 4.

2.4 Discussion

Although we equivalently derive both upper bounds and lower bounds of robust overfitting bias, there remains a gap in finding whether the lower and upper bounds can match up to a constant factor. In non-robust setting, it is proven that the √ log n factor in upper bounds can be removed for k = 1 [6]. It is interesting to ask if this result holds for adversarial robust setting and, more ambitiously, for all k. We leave it as an open question.

3 Proofs of main results

To make the proofs more readable, we introduce the notion of corrupted hypothesis [28].

̸

Consider a given hypothesis f : X → Y . A labeled adversarial sample (˜ x, y ) is classified correctly if ˜ x ∈ f -1 ( y ) . A labeled example ( x, y ) is classified correctly if U ( x ) ⊂ f -1 ( y ) . Let ˜ Y = Y ∪ {⊥} , where ⊥ is the special 'always wrong' output, i.e. y = ⊥ for every y ∈ ˜ Y . We define the mapping κ U : Y X → ˜ Y X :

The corrupted set of hypotheses induced by perturbation U is then defined by ˜ H = { κ U ( f ) : f ∈ H} .

3.1 Upper bounds

The upper bounds relies on the following theorem, showing in a probabilistic view that finding a classifier in some given hypothesis class with desired robust accuracy requires learning many bits about the test labeling.

Theorem 2. Let m,n,k be positive integers and D n m = D n X × µ n m , where µ n m denotes the uniform distribution over [ m ] n . Let H ⊂ Y X be a hypothesis class and U be an perturbation. Then there exists a constant C H ≤ 1 satisfying: For every H -based k -query algorithm A , δ > 0 , b = k ln( n +1)1 + ln(1 /δ ) , we have

and

Proof. For any fixed hypothesis f, denote κ U ( f ) by ˜ f. By the definition of corrupted hypotheses, for every i ∈ [ n ] ,

̸

̸

̸

We observe that Pr { ˜ f ( x i ) = ⊥} is a constant related to f, let C ( f ) ≜ Pr { ˜ f ( x i ) = ⊥} , since Pr { ˜ f ( x i ) = c i | ˜ f ( x i ) = ⊥} = 0 , we have

̸

̸

̸

This implies that ✶ {∀ x ′ ∈ U ( x i ) , f ( x ′ ) = c i } is a Bernoulli random variables with bias C ( f ) m . By the Chernoff bound, for any fixed f,

Denote max f ∈H C ( f ) by C H , then

holds for every f. Consider the execution of A with responses of robust accuracy fixed to some sequence of values α = ( α 1 , . . . , α k ) ∈ { 0 , 1 /n, . . . , 1 } k . Denote the resulting predictor by A α , its output distribution is fixed, hence by Eq.(1), we have

Denote the set of all possible values of α by V. For every test set S, the accuracy oracle outputs some responses in V. Therefore,

Now if k ln( n +1)+ln(1 /δ ) n ≥ 1 m , then by definition of b , 2 b n ≥ 2 m . It follows that mϵ ≥ 2 ≥ 2 C H , hence mnϵ 2 2 C H + mϵ ≥ nϵ 2 and

If k ln( n +1)+ln(1 /δ ) n < 1 m , in this case 2 √ b nm ≤ 2 m . We obtain that mϵ < 2 , thus mnϵ 2 2 C H + mϵ ≥ mnϵ 2 4 and

and we complete the proof.

The corollary below follows immediately from Theorem 2, which gives the upper bounds of h U ( k, n, m ) .

Corollary 1. Let m,n,k be positive integers and U be a perturbation, then

and

Proof. Denote Acc U ( f ; S ) ≜ X ∈ [0 , 1] . Substitute δ = 1 /n in Theorem 2 and notice that C H ≤ 1 for any hypothesis class H , hence for k < n m (log( n +1)+log( n )) we have

Let c = 1 m +2 √ ( k +1)log( n +1) nm , it remains to show E X ≤ c +1 /n. It is trivial for c ≥ 1 . For the case that c < 1 , let P X be the probability distribution of X, by the definition of expectation,

where we use the fact that ∫ 1 c dP X ≤ 1 /n by Eq.(2) in the last step. The case of k ≥ n m (log( n +1)+log( n )) can be proved using similar arguments, and we complete the proof.

3.2 Lower bounds

The lower bounds of h U ( k, n, m ) are derived from two designed algorithms, namely A small and A big ( C ) . A small is divided into two cases based on whether k = 1 , and the precise analysis is presented in Theorem 3. A big ( C ) accepts a parameter C for calculating an intermediate variable, and we gain our Theorem 4 by setting C = Φ D X ( n ) . Finally, the lower bounds of h U ( k, n, m ) follow from

To simplify the expression, we first introduce some definitions. For each X S ∈ X n , define

̸

and

where U (Γ S ) = {U ( x ) | x ∈ Γ S } and P D X is the distribution function of D X .

With these definitions, we present that:

Algorithm 1 A small ( k = 1 )

Let f 1 ( x ) be the all one query, i.e. f 1 ( x ) = 1 , ∀ x ∈ X .

Output ˆ f that

Theorem 3. Let n ≥ m and D n m = D n X × µ n m . Then for 1 ≤ k ≤ 1 + n/ 2 m,

Proof. Case: k = 1 : For l ∈ { 1 , . . . , m } , let N l be the number of examples with label l. Since the labels are uniformly distributed and ˆ f is always a constant predictor, ( N 1 , . . . , N m ) follows a multinomial distribution with parameters ( n ; 1 m , . . . , 1 m ) . Then by the construction of ˆ f, the number of robustly and correctly predicted labels is then given by N 1 · ✶ { N 1 ≥ n/m } + N 2 · ✶ { N 1 < n/m } , and the expected robust accuracy is given by

We conclude that

by citing the following lemma:

Lemma 1 (Lemma 4 in [6].) . Let n ≥ m ≥ 2 . If ( N 1 , . . . , N m ) is distributed according to a multinomial distribution with parameters ( n ; 1 m , . . . , 1 m ) , then

̸

Case: k > 1 : For a fixed S, and hence, a fixed ˆ f = A small ( S ) , for each l ∈ { 1 , . . . , m, ⊥} , let N S i,l be the number of examples in B i s.t. κ U ( ˆ f )( x ) = l. Since whether κ U ( ˆ f )( x ) equals to ' ⊥ ' is only depends on the distribution of x , we have that ( N S i, 1 , . . . , N S i,m , N S i, ⊥ ) follows a multinomial distribution with parameters ( | B i | ; C ( S ) m , . . . , C ( S ) m , 1 -C ( S )) , where C ( S ) = Pr { κ U ( ˆ f )( x ) = ⊥} .

Algorithm 2 A small ( k > 1 )

Divide [ n ] into k -1 blocks B 1 , . . . , B k -1 such that n k -1 ≤ | B i | ≤ n 2( k -1) for each i.

For 1 ≤ i ≤ k, define query f i satisfying:

for x j ∈ X S and for x / ∈ X S .

Output ˆ f, it predicts x j by the following:

and predicts the rest of x by ˆ f ( x ) = ˆ f ( x j ′ ) , where j ′ = arg min i ∈ [ n ] d p ( x, x j ) .

Then our predictions robustly and correctly predicts max { N S i, 1 , N S i, 2 } examples in B i and hence h U ( A ; S ) = 1 n ∑ k -1 i =1 max { N S i, 1 , N S i, 2 } . We will show it later that

By summing over the blocks, we can lower bound the expected total number of robustly correct predictions made by A small :

To obtain a lower bound of C ( S ) that is independent of the choice of S, for each S we define g S satisfying:

and let H n = { g S : S ∈ (supp( D X ) ×Y ) n } . It is easy to see that A small ( S ) ∈ H n for all S. Hence by the definition of Φ D X ( n ) , we have

̸

We conclude that

Normalizing by n proves the desired result. It remains to lower bound E [max { N S i, 1 , N S i, 2 } ] . Since N S i, 1 and N S i, 2 follow a binominal distribution with parameters ( | B i | ; C ( S ) m ) , E [ N S i, 1 + N S i, 2 ] = 2 C ( S ) | B i | m for all S. Let N ′ be an independent copy

of N S i, 2 . N S i, 1 and N ′ are negatively correlated, hence

For all S.

Then we conclude that

which completes the proof.

We then present A big ( C ) and the theoretical analysis for k = Ω( n/m ) .

Algorithm 3 A big ( C )

Let t := 1 + Ck 9 ln m

Define query f 1 , . . . , f k satisfying:

(a) For 1 ≤ j ≤ t, f 1 ( x j ) , . . . , f k ( x j ) are uniformly chosen from all sequences in [ m ] k that have each element in [ m ] appearing exactly k/m times.

(b) For j > t, f i ( x j ) = ⊥ for all i = 1 , . . . , k.

(c) For x / ∈ X S , f i ( x ) = f i ( x j ′ ) for each i ∈ [ k ] , where j ′ = arg min i ∈ [ n ] d p ( x, x j ) .

Output ˆ f such that:

and predicts the rest of x by ˆ f ( x ) = ˆ f ( x j ′ ) , where j ′ = arg min i ∈ [ n ] d p ( x, x j ) .

Theorem 4. Let k > 18 m log m Φ DX ( n ) and let D n m = D n X × µ n m ,

Proof. For l ∈ [ m ] , let A l be the total number of robustly and correctly predicted examples by all the queries that predict the first examples as ' l ', i.e.

where W 0 ≤ m is the number of queries that predict the whole perturbation set U ( x 1 ) as ' y 1 '.

Let

̸

then for l = y 1 ,

̸

Since ✶ { f i ( x ′ ) = y j , ∀ x ′ ∈ U ( x j ) } are independent for j = j ′ , and are negatively associated across i for any fixed j, therefore, by the Chernoff bound, it satisfies that for ϵ < 1

Suppose ϵ satisfies that ϵ E [ M l ] ≤ k 2 m and ϵ 2 3 E [ M l ] ≥ 3 log m, then by Eq. (3) and the union bound

̸

and with probability at least 3 / 4 , ˆ f ( x 1 ) = y 1 .

We now derive the bounds on ϵ. To this end, we first need to bound E [ M l ] . Define H n as that in the proof of Theorem 3. Let ˜ H n denotes the corrupted set of H n . For each ˜ g ∈ ˜ H t , define ˜ g ⊥ : X → ˜ Y that takes value ' ⊥ ' on U ( x j ) for j = t +1 , . . . , n and coincides with ˜ g otherwise, that is,

Let G n be the set which contains all such ˜ g ⊥ . It is easy to see that κ U ( f 1 ) , . . . , κ U ( f k ) ∈ G n . Note that min g ∈G n Pr { g ( x ) = ⊥} = Φ D X ( n ) , we have

̸

Now for each l by the linearity of expectations,

Thus ϵ E [ M l ] holds for

and ϵ 2 3 E [ M l ] ≥ 3 log m holds for ϵ ≥ √ 9 log m · m 2 Φ DX ( n )( t -1) k . Therefore, we can find a suitable ϵ whenever

̸

If we choose t = 1 + Φ DX ( n ) k 36 log m and k > 18 Φ DX ( n ) m log m, the aforementioned conditions hold. Note that Pr { κ U ( ˆ f ) = ⊥} ≥ Φ D X ( n ) , hence the expected number of robustly and correctly predicted labels is at least

which completes the proof.

4 Conclusion

In this work, we study the overfitting bias in the context of robust multiclass learning. We formally define the adaptive algorithms in an adversarial setting and analyze the average case performance that can be achieved by an adaptive algorithm. Upper bounds and lower bounds are both derived, and are matching within logarithmic factors when the number of test samples and distribution of data features are fixed.

References

Leslie Rice, Eric Wong, and J. Zico Kolter. Overfitting in adversarially robust deep learning. In ICML , volume 119, pages 8093-8104, 2020.

Avrim Blum and Moritz Hardt. The ladder: A reliable leaderboard for machine learning competitions. In ICML , volume 37 of JMLR Workshop and Conference Proceedings , pages 1006-1014, 2015.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML , volume 97 of Proceedings of Machine Learning Research , pages 5389-5400, 2019.

Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Leon Roth. Preserving statistical validity in adaptive data analysis. In STOC , pages 117-126, 2015.

Vitaly Feldman, Roy Frostig, and Moritz Hardt. The advantages of multiple classes for reducing overfitting from test set reuse. In ICML , volume 97, pages 1892-1900, 2019.

Jayadev Acharya and Ananda Theertha Suresh. Optimal multiclass overfitting by sequence reconstruction from hamming queries. In ALT , volume 117 of Proceedings of Machine Learning Research , pages 3-21, 2020.

Paul Erdos and Alfréd Rényi. On two problems of information theory. Magyar Tud. Akad. Mat. Kutató Int. Közl , 8:229-243, 1963.

Vasek Chvátal. Mastermind. Combinatorica , 3(3):325-329, 1983.

Nader H. Bshouty. Optimal algorithms for the coin weighing problem with a spring scale. In COLT , 2009.

Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. The reusable holdout: Preserving validity in adaptive data analysis. Science , 349(6248):636-638, 2015.

Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. Generalization in adaptive data analysis and holdout reuse. In NeurIPS , pages 2350-2358, 2015.

Raef Bassily, Kobbi Nissim, Adam D. Smith, Thomas Steinke, Uri Stemmer, and Jonathan R. Ullman. Algorithmic stability for adaptive data analysis. In STOC , pages 1046-1059, 2016.

Vitaly Feldman, Roy Frostig, and Moritz Hardt. Open problem: How fast can a multiclass test set be overfit? In COLT , volume 99 of Proceedings of Machine Learning Research , pages 3185-3189, 2019.

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR , 2014.

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR , 2015.

Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In ICML , volume 202 of Proceedings of Machine Learning Research , pages 36246-36263, 2023.

Dong Yin, Kannan Ramchandran, and Peter L. Bartlett. Rademacher complexity for adversarially robust generalization. In ICML , volume 97 of Proceedings of Machine Learning Research , pages 7085-7094, 2019.

Omar Montasser, Steve Hanneke, and Nathan Srebro. VC classes are adversarially robustly learnable, but only improperly. In COLT , volume 99 of Proceedings of Machine Learning Research , pages 2512-2530, 2019.

Jingyuan Xu and Weiwei Liu. On robust multiclass learnability. In NeurIPS , 2022.

Zhengyu Zhou and Weiwei Liu. Sample complexity for distributionally robust learning under chi-square divergence. Journal of Machine Learning Research , 24(230):1-27, 2023.

Zekai Wang and Weiwei Liu. Robustness verification for contrastive learning. In ICML , volume 162 of Proceedings of Machine Learning Research , pages 22865-22883, 2022.

Xin Zou and Weiwei Liu. Generalization bounds for adversarial contrastive learning. Journal of Machine Learning Research , 24:114:1-114:54, 2023.

Hanshu Yan, Jiawei Du, Vincent Y. F. Tan, and Jiashi Feng. On robustness of neural ordinary differential equations. In ICLR , 2020.

Xiyuan Li, Zou Xin, and Weiwei Liu. Defending against adversarial attacks via neural dynamic system. In NeurIPS , 2022.

Xinsong Ma, Zekai Wang, and Weiwei Liu. On the tradeoff between robustness and fairness. In NeurIPS , 2022.

Boqi Li and Weiwei Liu. WAT: improve the worst-class robustness in adversarial training. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, AAAI , pages 14982-14990, 2023.

Yao-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Ruslan Salakhutdinov, and Kamalika Chaudhuri. A closer look at accuracy vs. robustness. In NeurIPS , 2020.

Daniel Cullina, Arjun Nitin Bhagoji, and Prateek Mittal. Pac-learning in the presence of adversaries. In NeurIPS , pages 228-239, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main worry regarding the excessive reuse of test datasets in machine learning is its potential to cause overfitting. The objective of this paper is to characterize the relationship between the amount of robust overfitting bias and three key factors: the number of classes (m), the number of robust accuracy queries (k), and the size of the test dataset (n). The main theoretical contributions of this paper consist of providing upper and lower bounds on the attainable robust overfitting bias in multiclass settings.\n\nSTRENGTHS:\nThe paper showcases exceptional writing skills, surpassing the average standard of presentation seen in many well-written and well-structured papers. The introduction successfully establishes a clear motivation and provides a satisfactory outline of the contributions. Although the section on related works is concise, it not only acknowledges a broad range of prior research efforts but also effectively highlights the key distinctions and limitations. Despite my personal unfamiliarity with several of the cited works and some of the presented topics, I never felt lost. This is mainly attributed to the \"Summary of our results\" section, which introduces the various notations used in the paper, provides a very clear formulation of the addressed problem, and offers a summary of the main results and the techniques employed in their proofs. The proofs of the various theoretical results are generally clear and sufficiently detailed. However, there are occasional passages that I did not understand easily and, in my opinion, require more elaboration. For instance, the usage of the Chernoff bound for Bernoulli random variables with bias could benefit from further explanation.\n\nWEAKNESSES:\nMy primary concern relates to the level of novelty and originality in the theoretical results presented in the paper, particularly concerning the techniques employed in the proofs. It is worth acknowledging that the paper serves as an extension of previous works by (Feldman et al., \"The advantages of multiple classes for reducing overfitting from test set reuse,\" ICML'19) and (Acharya & Suresh, \"Optimal multiclass overfitting by sequence reconstruction from Hamming queries,\" ALT'20) within an adversarial setting. However, it is important to note that the proofs in this paper largely follow a similar approach to those in the aforementioned papers, and the extension to the adversarial setting does not significantly contribute to the technical aspect.\n\nI have observed several errors in the paper (which I believe are oversights) that can be easily noticed but may lead to a misunderstanding of the contributions. One specific example is found in line 1 of Algorithm 2, where the division of n into (k-1) blocks should be denoted as (B_1...B_(k-1)) instead of (B_1, ..., b_k). It is crucial to maintain a consistent notation for the blocks using a capital B, and it should be noted that the last block is indexed as (k-1). Furthermore, on the same line, the upper and lower bounds of |B_i| need to be reversed due to the condition n > k-1. The correct bounds should be $\\frac{n}{2(k-1)} \\leq |B_i| \\leq \\frac{n}{k-1}$. These errors should be rectified to ensure the accuracy and clarity of the paper.\n\nAnother aspect that I find regrettable is the lack of discussion or interpretation regarding the various bounds presented in the paper. This absence leaves a gap in the understanding of the implications and significance of these bounds. It would have been valuable to explore the practical implications of the bounds, their relationship to existing theories or frameworks, and any potential limitations or extensions that arise from them. Providing such a discussion would have enhanced the overall comprehensiveness and depth of the paper.\n\nQUESTIONS:\nSee the above Flaws part.\n\nLIMITATIONS:\nNo potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper generalizes the framework of perfect reconstruction to the adversarial setting and studies how much a k-query algorithm can overfit the test set in the adversarial setting. Upper bounds and lower bounds are derived, which match in terms of the number of classes m and the number of queries k modulo logarithmic factors when the number of test examples n and the distribution are fixed.\n\nSTRENGTHS:\nThis paper gives a careful analysis that generalizes both the upper bound and the lower bound on how much an algorithm can overfit the test set in non-robust setting to the adversarial setting. The results are novel.\n\nWEAKNESSES:\n1. The dependence of the lower bound on the number of test examples n does not have a closed form and thus the question asked in the abstract is not fully solved.\n\n2. It is not clear what role the perturbation set U plays in the bounds. In the adversarial setting, other than the parameters m, k and n, the radius r of the perturbation ball is also an important parameter and it is usually more informative if we know how the bounds are affected by allowing stronger attacks, i.e. a larger r. Otherwise, some intuitions on why the perturbation size is irrelevant here would be helpful.\n\n3. This paper is in general not very well-written. For example, the term robust overfitting is confusing here: in line 9, robust overfitting [1] refers to the phenomenon that the robust test error can increase during adversarial training, but later on, e.g. in line 21 and 81, the term is used to describe the overfitting to the test set with k-query algorithms; there are not many detailed discussions on the motivation of the studied problem or what the derived bounds suggest such as how each parameter affects the bound and if robust multiclass classification requires benchmark datasets different from standard classification.\n\nQUESTIONS:\nSee Weaknesses.\n\nminors:\n1. In line 66, should it be h_U(A) - 1/m instead?\n2. In line 107, the explanation after 'i.e.' is confusing.\n3. In Algorithm 1, the definition of the output is not written in a very nice way. Probably move $\\forall x \\in X$ outside.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors consider the following question: Given the number of classes m, the number of robust accuracy queries k, and the number of test examples in the dataset with size n, how much can adaptive algorithms robustly overfit the test dataset? They solve this problem by giving upper and lower bounds of the robust overfitting accuracy in multiclass classification problems.\n\nSTRENGTHS:\n1. Robust classification and adversary perturbation are valuable topics in the learning theory community.\n2. Some adaptive algorithms are proposed and could be useful in practice.\n\nWEAKNESSES:\nMajor\n1. The paper structure is not very clear. The algorithms are defined after the main results. It's very confusing.\n2. Line 56-69: The definition of k-query is not reader-friendly. It's better to have some examples and figures.\n3. Line 63-65: The assumption that the labels are uniformly distributed is very restricted and not general. It makes the study has limited practical and theoretical contribution. \n4. The gap between the upper and lower bound is large, which makes the two bounds have limited theoretical and practical contributions.\n5. In Lines 49-52, the perturbation is defined with radius r. However, in all main results, they don't contain r, and have no restriction on r. The result would be different between a tiny and a huge r.\n\nMinor\n1. The literature section needs a clearer analysis of the relationship among adaptive algorithms, overfitting, robustness, and multi-classification. In addition, the motivation is not fully explained.\n2. The title \"Characterization of Overfitting in Robust Multiclass Classification\" is too broad. It's better to be more specific with the adaptive algorithms and adversarial settings.\n3. What's mathmatical definition for \\O, \\Omega, and \\Theta? They are not defined in the paper.\n4. The main result, Theorem 1 (lines 74-76), is \nlabeled as informal. In addition, it's not proven and not reader-friendly.\n5. The paper lacks practical experiments to support the main results.\n\nQUESTIONS:\n1.What's mathmatical definition for \\O, \\Omega, and \\Theta? They are not defined in the paper.\n\n2. Line 56-69: The definition of k-query is not reader-friendly. It's better to have some examples and figures. Could you provide a detailed explanation?\n\n3. In Lines 49-52, the perturbation is defined with radius r. However, in all main results, they don't contain r, and have no restriction on r. The result would be different between a tiny and a huge r. Could you explain the relation between accuracy and r?\n\nLIMITATIONS:\nThis paper doesn't include limitations and the negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of learning from data while being robust to the possible transformations of these data. A common practice in machine learning is to split data into a training and a test set. The latter is a holdout to evaluate the performance of the algorithm. However, recent studies have shown that reusing many times this holdout set leads to overfitting in non-robust settings. This paper questions the problem of adaptivity in an adversarial/robust setting.\n\nSTRENGTHS:\nThe authors present a very theoretical work on robustly overfitting the test set of adaptative algorithms. They show an excellent understanding of the theory behind this problem. Remarkably, the upper bounds in the Theorem 1 match the ones of the literature when $\\mathcal{U} = \\mathcal{I}$ (no transformation of the set, i.e. no adversarial setting). Furthermore, for a fixed size of $S$, the test set, whose features are i.i.d. according to a fixed distribution, the upper and lower bounds match up to a logarithmic factor.\n\nWEAKNESSES:\nThe paper is hard to follow for someone unfamiliar with the topic like myself. The Neurips conference gathers people from a broad audience, and this must be taken into account. In particular, no experimental part in the paper limits its impact on the community. Some numerical experiments could help understand how this work can be applied and its limits for future directions. Adding a toy numerical example could also motivate us why we should care about the question the authors ask in the abstract.\n\nQUESTIONS:\nSuggestions:\n- recall what are $O, \\Omega$ and $\\Theta$\n- define $\\widetilde{O}, \\widetilde{\\Omega}, \\widetilde{\\Theta}$ even if it seems to be trivial for the authors,\n- add an experimental section with experiments at least on simulated data and, better, on real datasets to illustrate the different theorems.\n\nQuestion:\n- are these $\\widetilde{O}, \\widetilde{\\Omega}, \\widetilde{\\theta}$ a classical tool in your community? Why not expose the full bound with the logarithm factor? Does it help the interpretation?\n\nLIMITATIONS:\nNo limitations are declared in the paper. The paper is theoretical and seems far from having a possible negative impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies bounds on the maximal difference between the robust accuracy of a classifier on the test set. Here, robustness means when each instance is allowed to move within a given radius within an $L_p$ ball, and the maximal difference is obtained with respect to $1/m$, where $m$ is the number of classes. \n\nThe classifier's accuracy is obtained through querying an algorithm, and lower/upper bounds are obtained in terms of the number of queries made, the number of elements in the test set and the number of classes.\n\nSTRENGTHS:\n* A concise paper that, for once, does not have tons of supplementary material. \n\n* A rather rigorous treatment of the problem, with the paper consisting almost entirely of providing the proofs of the results. I think the results presented are correct (but this is not in my main area of expertise, so I could have missed something, and it would be necessary to be checked by a more expert reviewer foxued on this area of research)\n\nWEAKNESSES:\n* Better motivation of the results through illustration/practical problem mention: the paper focuses on providing a rigorous bound proof for a particular setting (robust overfitting), and assumes that the reader is convinced of the importance of the problem. In this sense it is quite focused, and it would have been nice for the reader to have at least some indication of the practical setting where this result could be used, or even better a little illustrative (synthetic) example of the situation one would like to deal with. This could be done, e.g., by dropping Lemma 1 proof if this one is available elsewhere (Lemma 4 in [6], according to the paper), as it takes almost one pages.\n\n* Clarifications about the assumptions and their limitations: in order to obtain the provided results, the paper makes some key assumptions whose limitations and need should be discussed in more details. Two main examples of this are the following: \n\n   * Well-separatedness of the classes is mentioned P3, L3 (note: it is not mentioned this way in reference [14]) seems equivalent to assume a dceterministic relationship between input and classes, with furthermore the requirement to be separated by \"large margins\", so basically assuming that one is working under a version space assumption. There would be a need to clarify that, to discuss how realistic this is, and how it can impact the provided results\n\n  * Uniformity of the labels: another assumptions is that the output labels are perfectly uniformly distributed. This is unrealistic in most practical settings, and it is not clear how the results would change in case of imbalanced classes (goign from a slight departure of this prior information to a large one).\n\nQUESTIONS:\nQuestions\n---\n\n(wrap up weakness points for some)\n\n* Would it be possible to provide a slight illustration of the discussed results, or to give pointers to practical uses of these results? \n\n* The idealistic assumptions made in the paper (well-separatedness, label uniformity) are rarely met in practice. It is not clear how limiting this is, would one apply the obtained bounds in a practical setting? \n\n* I did not find in the paper a discussion about the quality of the provided bounds: can they expected to be tight? How useful are they likely to be in practice? \n\nSuggestions\n---\n\n* Some typos remain in the paper (e.g., in Theorem 2: an perturbation, $b=k ln(n+1)1$, L171: \"we will show it later than\")\n\n* Some parts of the proofs and technical aspects could be clarified by adding some steps that I missed while reading it. As examples:\n   * P2, L39: it may be obvious, but what is the \"closure\" here (are we speaking about a specific closure operator?)\n   * P5, L131: the derivation of $m\\epsilon \\geq 2 \\geq 2 \\mathcal{C}_\\mathcal{H}$ is not immediate to me, maybe add a step?\n   * P6, L161-L162: I do not see how we can obtain the numbers of correctly predicted labels only depend on the $N_m$, and also why it should only depends on $N_1,N_2$?\n\nLIMITATIONS:\nSee weaknesses and questions comments about the initial assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-17"}

comments: SUMMARY:
The authors present near-matching lower and upper bounds for the robust accuracy of an adaptive algorithm having a budget of k accesses to the oracle of accuracy on a test set of size n and m classes.

STRENGTHS:
The proof is really clear and pedagogical. The related work is clear.

WEAKNESSES:
The problem is not well sold nor introduced. I had to look at the problem formulation to understand what the authors were intending to do. While related work is abundant, more space can be used for explaining what are the goals of the paper for new readers. And the conclusion could be more than just a summary, starting future work.

Also, as the authors are humble, they are borrowing tools from several existing proofs so it is not clear whether this is a breakthrough or incremental work. Still, it is nicely put.

Minor comments:
"binominal" => binomial

QUESTIONS:
No

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-25