Title: Improved Sample Complexity for Multiclass PAC Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multiclass learning

PAC learning

Statistical learning

List learning
Abstract: We aim to understand the optimal PAC sample complexity in multiclass learning. While finiteness of the Daniely-Shalev-Shwartz (DS) dimension has been shown to characterize the PAC learnability of a concept class [Brukhim, Carmon, Dinur, Moran, and Yehudayoff, 2022], there exist polylog factor gaps in the leading term of the sample complexity. In this paper, we reduce the gap in terms of the dependence on the error parameter to a single log factor and also propose two possible routes towards completely resolving the optimal sample complexity, each based on a key open question we formulate: one concerning list learning with bounded list size, the other concerning a new type of shifting for multiclass concept classes. We prove that a positive answer to either of the two questions would completely resolve the optimal sample complexity up to log factors of the DS dimension.

Improved Sample Complexity for Multiclass PAC Learning

Abstract

We aim to understand the optimal PAC sample complexity in multiclass learning. While finiteness of the Daniely-Shalev-Shwartz (DS) dimension has been shown to characterize the PAC learnability of a concept class [Brukhim, Carmon, Dinur, Moran, and Yehudayoff, 2022], there exist polylog factor gaps in the leading term of the sample complexity. In this paper, we reduce the gap in terms of the dependence on the error parameter to a single log factor and also propose two possible routes towards completely resolving the optimal sample complexity, each based on a key open question we formulate: one concerning list learning with bounded list size, the other concerning a new type of shifting for multiclass concept classes. We prove that a positive answer to either of the two questions would completely resolve the optimal sample complexity up to log factors of the DS dimension.

1 Introduction

Multiclass learning refers to the problem of classifying an input feature from a set (feature space) X to a label in a set (label space) Y with | Y | ą 2 ( Y can be infinite) [Natarajan, 1989, Ben-David et al., 1995, Daniely and Shalev-Shwartz, 2014, Brukhim et al., 2022]. When | Y | ' 2 , the problem is known as binary classification. Multiclass learning has wide applications to various tasks in machine learning including image classification [Rawat and Wang, 2017], natural language processing [Kowsari et al., 2019], tissue classification [Li et al., 2004], etc. For theoretical analysis of multiclass learning, a probabilistic setting is typically assumed, where all the feature-label pairs in the training sequence are assumed to be independent and identically distributed (iid) samples from some distribution P over X ˆ Y . Then, the objective of the learner is to minimize the error rate of the output classifier under the distribution P . A basic framework in the probabilistic setting is Probably Approximately Correct (PAC) learning [Valiant, 1984]. Though the characterization of PAC learnability of a binary concept class with the finiteness of its Vapnik-Chervonenkis (VC) dimension has been proved by Vapnik and Chervonenkis [1968], the characterization of the multiclass PAC learnability remained open until Brukhim et al. [2022] showed the equivalence between the PAC learnability of a concept class and the finiteness of its Daniely-Shalev-Shwartz (DS) dimension ( dim , see Definition 1.4) [Daniely and Shalev-Shwartz, 2014] instead of Natarajan dimension or graph dimension.

However, the problem of establishing the optimal sample complexity or error rate (see Section 1.1 for formal definitions) for multiclass learning remains unsolved. For binary concept classes, Hanneke [2016] showed that the sample complexity is in Θ pp d ` log p 1 { δ qq{ ε q where d is the VC dimension of the concept class. Since DS dimension and VC dimension coincide for binary concept classes, it is natural to ask if the sample complexity of a concept class H Ď Y X for | Y | ą 2 is also in

Θ pp d ` log p 1 { δ qq{ ε q where d ' dim p H q is the DS dimension of H . In terms of the upper bound, it asks if there exists a multiclass learner whose worst case error rate is in O pp d ` log p 1 { δ qq{ n q with probability at least 1 ´ δ , where n denotes the size of the training sequence. However, on the one hand, an explicit proof of the Ω pp d ` log p 1 { δ qq{ ε q lower bound on the sample complexity is still missing in the literature. On the other hand, the current best upper bound on worst case error rate to our knowledge is O ` p d 3 { 2 log p d q` d log p log p n qqq log 2 p n q` log p 1 { δ q n ˘ [Brukhim et al., 2022], which differs from the conjectured rate by the factor p log p log p n qq ` ? d log p d qq log 2 p n q .

In this paper, we step forward towards improved sample complexity and error rate in multiclass learning. As the concept class is fixed and the sample size increases during an online learning process, we mainly focus on improving the error rate in terms of the sample size n . Specifically, for a concept class H Ď Y X with dim p H q ' d , we prove an Ω pp d ` log p 1 { δ qq{ ε q lower bound on its sample complexity and construct a multiclass learner whose worst case error rate is O pp d 3 { 2 log p d q log p n q ` log p 1 { δ qq{ n q with probability at least 1 ´ δ , which implies that the sample complexity of H is O ` d 3 { 2 log p d q log p d { ε q` log p 1 { δ q ε ˘ . Our results greatly narrow the gap between the upper and lower bounds of the sample complexity and the error rate. The dependence of the upper bound of the error rate on the sample size has also been improved from O p log p log p n qq log 2 p n q{ n q to O p log p n q{ n q (treating d as a constant). The multiclass learner we construct builds upon a list learner which predicts a list of labels for the test point (see Section 2.1 for a detailed introduction to list learning). Actually, we prove a reduction from multiclass learning to list learning and upper bound the error rate of the constructed multiclass learner with some function of the list size and the expected error rate of the list learner (the probability of excluding the true label in the predicted list). Moreover, the upper bound indicates that a list learner with size independent of n and expected error rate scaling linearly in 1 { n in terms of the sample size n would imply an O p 1 { n q error rate (treating d as a constant). We leave the construction of such list learners an open question.

Furthermore, we also explore an alternative combinatorial approach towards improved sample complexity in multiclass learning. For a concept class, we can define a hypergraph called the one-inclusion graph [Haussler et al., 1994] on its projection to a finite sequence of features (see Section 1.1 for definitions). Then, informally speaking, the 'density' (defined through the average degree of the one-inclusion graph) of a concept class can be used to upper bound the error rate of multiclass learning [Daniely and Shalev-Shwartz, 2014, Aden-Ali et al., 2023]. Specifically, if we can upper bound the density of any concept class H by a multiple of its DS dimension, then the sample complexity is in O pp dim p H q ` log p 1 { δ qq{ ε q , which matches the lower bound we prove. Thus, a proof of the above upper bound directly leads to a Θ pp dim p H q ` log p 1 { δ qq{ ε q sample complexity for multiclass learning. When dim p H q ' 1 , we successfully prove the Θ p log p 1 { δ q{ ε q sample complexity in Theorem 3.2. For general concept classes, we develop a technique named 'pivot shifting' similar to the shifting operator [Haussler, 1995] on concept classes. We show that if a pivot shifting does not increase the DS dimension of a concept class, then its density is upper bounded by twice the DS dimension. We leave the impact of pivot shifting on DS dimension as another open question.

Throughout the paper, we use N to denote the set of positive integers. For any n P N , we define r n s : ' t 1 , . . . , n u . For any sets X , Y , sequence x ' p x 1 , . . . , x n q P X n , and function f P Y X , we define the subsequence x ´ i : ' p x 1 , . . . , x i ´ 1 , x i ` 1 , . . . , x n q for i P r n s and f | x : ' p f p x 1 q , . . . , f p x n qq . The projection of a set F Ď Y X to x is defined as F | x : ' t f | x : f P F u Ď Y n .

Outline In Section 1.1, we introduce the problem of multiclass learning and review some existing results. In Section 1.2, we summarize the key points of our theoretical results. In Section 2, we introduce list learning, present the reduction from multiclass learning to list learning, and improve the sample complexity upper bound of multiclass learning via this reduction together with a boosting technique for list learners. In Section 3, we prove the optimal sample complexity for classes of DS dimension 1, introduce the intuition and the definition of 'pivot shifting', and demonstrate its potential application to the proof of the optimal sample complexity of multiclass learning.

1.1 Multiclass learning

In this section, we formally introduce the problem of multiclass learning [Valiant, 1984]. For any distribution P over X ˆ Y , the error rate of a classifier h P Y X under P is defined as

In this paper, we focus on realizable distributions: for a concept class H Ď Y X , a distribution P over X ˆ Y is H -realizable if inf h P H er P p h q ' 0 . Let RE p H q denote the set of H -realizable distributions. Besides, pp x i , y i qq n i ' 1 P p X ˆ Y q n is H -realizable if D h P H such that y i ' h p x i q , @ i P r n s .

Definition 1.1 (Multiclass learner) . A multiclass learner (or a learner) A is an algorithm which given a sequence s P Y 8 n ' 0 p X ˆ Y q n and a concept class H Ď Y X , outputs a classifier A p s , H q P Y X .

Then, we can define multiclass PAC learning as follows.

Definition 1.2 (Multiclass PAC learning) . For any concept class H Ď Y X , the (PAC) sample complexity M A , H : p 0 , 1 q 2 Ñ N of a multiclass learner A is a mapping from p ε, δ q P p 0 , 1 q 2 to the smallest positive integer such that for any m ě M A , H p ε, δ q and any distribution P P RE p H q , P S ' P n p er P p A p S, H qq ą ε q ď δ , and we define M A , H p ε, δ q ' 8 if no such integer exists. We say H is PAC learnable by A if M A , H p ε, δ q ă 8 for all p ε, δ q P p 0 , 1 q 2 . The (PAC) sample complexity of H is defined as M H p ε, δ q : ' inf A M A , H p ε, δ q for any p ε, δ q P p 0 , 1 q 2 .

Sometimes it is easier to analyze the expected error rate

for a learner A and distribution P over X ˆ Y , or transductive error rate

We further define ε A , H : ' sup P P RE p H q ε A , H ,P , ε H : ' inf A ε A , H , and ε H , trans : ' inf A ε A , H , trans . By a leave-one-out argument [Brukhim et al., 2022, Fact 14], we observe that ε A , H ď ε A , H , trans . Aden-Ali et al. [2023, Theorem 2.1] upper bounded the high probability error rate using the transductive error rate, which leads to a guarantee on PAC sample complexity. Based on their result, we prove the same upper bound up to a multiplicative constant on the high probability error rate using the expected error rate in Theorem 2.6.

Next, we define pseudo-cubes and DS dimensions of concept classes. Here, we also present their extensions to the setting of k -list learning for future reference in Section 2.1.

Definition 1.3 (Pseudo-cube and k -pseudo-cube) . For any d, k P N , a class H Ď Y d is called a k -pseudo-cube of dimension d if it is non-empty, finite, and for every h P H and i P r d s , there exist at least k i -neighbors of h in H , where g is an i -neighbor of h if g p i q ‰ h p i q and g p j q ' h p j q for all j P r d szt i u . A pseudo-cube of dimension d is a 1 -pseudo-cube of dimension d .

Definition 1.4 (DS dimension and k -DS dimension, Charikar and Pabbaraju 2023) . For any d, k P N , we say x P X d is k -DS shattered by H Ď Y X if H | x contains a d -dimensional k -pseudo-cube. The k -DS dimension dim k p H q of H is the maximum size of a k -DS shattered sequence. We say x is DS shattered by H if it is 1-DS shattered by H . The DS dimension dim p H q of H is defined as dim 1 p H q .

Now, we introduce some existing results in multiclass learning. Brukhim et al. [2022] proved that a class H Ď Y X is PAC learnable if and only if d : ' dim p H q ă 8 , and there exists a multiclass learner A which for any P P RE p H q , δ P p 0 , 1 q , n P N , and S ' P n , satisfies that with probability at least 1 ´ δ ,

which is also the best upper bound before this paper to our knowledge. In terms of lower bound, it follows from Charikar and Pabbaraju [2023, Theorem 6] that ε H p n q ' Ω p d { n q . Thus, the current upper and lower bounds of the expected error rate does not match. Moreover, a potentially sharp lower bound on the sample complexity M H is still missing.

The learner A in Brukhim et al. [2022] relies on orienting the one-inclusion graphs defined below as a building block.

Definition 1.5 (One-inclusion graph, Haussler et al. 1994) . The one-inclusion graph (OIG) of H Ď Y n for n P N is a hypergraph G p H q ' p H,E q where H is the vertex-set and E denotes the edge-set defined as follows. For any i P r n s and f : r n szt i u Ñ Y , we define the set e i,f : ' t h P H : h p j q ' f p j q , @ j P r n szt i uu . Then, the edge-set is defined as

For any p e i,f , i q P E and h P H , we say h P p e i,f , i q if h P e i,f and the size of the edge is |p e i,f , i q| : ' | e i,f | .

Typically, we consider the one-inclusion graph of the projection of a concept class H Ď Y X to a sequence x P X n with n P N , i.e., G p H | x q . The 'density' of H discussed in Section 3 is defined via the 'maximal average degree' (defined below) of the hypergraph G p H | x q .

Definition 1.6 (Degree and average degree) . For any hypergraph G ' p V, E q and v P V , we define the degree of v in G to be deg p v ; G q : ' |t e P E : v P e, | e | ě 2 u| . When the underlying graph is clear in the context, we simply write deg p v q in abbreviation. If | V | ă 8 , we can define the average degree and average out-degree of G to be

For general V , we can define the maximal average degree of G to be md p G q : ' sup U Ď V : | U |ă8 avgdeg p G r U sq , where G r U s ' p U, E r U sq denotes the induced hypergraph of G on U Ď V with E r U s : ' t e X U : e P E,e X U ‰ Hu .

Note that for finite graphs, the average out-degree does not depend on the choice of orientation on G . Moreover, since | e | ě 2 for all e P E n , we have

Now, we can define the density of a concept class as follows.

Definition 1.7. The density of H Ď Y X is defined as µ H p m q : ' sup x P X m md p G p H | x qq , @ m P N .

1.2 Main results

In this section, we summarize the key points of our theoretical results. The full versions of some results are stated in Section 2 and 3. We first need the following definition to rule out trivial concept classes for which one training point suffices to achieve zero error rate under any realizable distribution.

Definition 1.8 (Nondegenerate concept class, Hanneke et al. 2023) . A concept class H P Y X is called nondegenerate if there exist h 1 , h 2 P H and x 0 , x 1 P X such that h 1 p x 0 q ' h 2 p x 0 q and h 1 p x 1 q ‰ h 2 p x 1 q . H is called degenerate if it is not nondegenerate.

Our main result on the multiclass PAC sample complexity is as follows.

Theorem 1.9 (Partial summary of Theorem 2.5 and 2.11) . For any nondegenerate concept class H Ď Y X with dim p H q ' d and any p ε, δ q P p 0 , 1 q 2 , we have

Our upper bound follows from a reduction to list learning and an improved sample complexity for list learning summarized below.

Theorem 1.10 (Informal summary of Theorem 2.7 and 2.10) . Assume that there exists a list learner which given a concept class H with dim p H q ' d and training sequence of size n P N outputs a menu of size p p H , n q with expected error rate upper bounded by β p H , n q{ n for some functions p and β nondecreasing in n . Then, there exists a multiclass learner whose error rate is

Moreover, there exists a list learner satisfying p p H , n q ' O ` p e ? d q ? d log p n q ˘ and β p H , n q ' O ` d 3 { 2 log p d q log p n q ˘ .

We refer readers to Section 2.1 for detailed definitions regarding list learning. Note that if p p H , n q and β p H , n q of some list learner is independent of n , there exists a multiclass learner with error rate linear in 1 { n . We leave the establishment of such list learners as Open Question 1.

In addition to the above approach, we propose an alternative route toward obtaining the conjectured Θ pp d ` log p 1 { δ qq{ ε q sample complexity, by directly bounding the average degrees of one-inclusion graphs. In particular, we show in Theorem 3.2 that any H with dim p H q ' 1 has M H p ε, δ q ' Θ p log p 1 { δ q{ ε q , which was not previously known. Moreover, we approach the general case via a new technique we call 'pivot shifting'. Specifically, we obtain the following result, which relies on an assumption on such pivot shifting. The verification of this assumption is left as Open Question 2.

Proposition 1.11 (Informal summary of the results in Section 3) . Assume that for any finite concept class, there exists a pivot shifting such that the DS dimension of the concept class does not increase after the pivot shifting, then we have M H p ε, δ q ' Θ pp dim p H q ` log p 1 { δ qq{ ε q for any H Ď Y X .

2 Multiclass learning via list learning

In this section, we prove a reduction from multiclass learning to list learning in Section 2.2. We improve the existing list learners using boosting in Section 2.3. Then, using a boosted list learner and the reduction, we improve the multiclass learning sample complexity upper bound in Section 2.4. We first present some definitions and results of list learning in Section 2.1.

2.1 List learning

In list learning, the menus defined below serve as classifiers in multiclass learning.

Definition 2.1 ( k -menu, Brukhim et al. 2022) . A menu of size k P N is a function µ : X Ñt Y Ď Y : | Y | ď k u . A 1 -menu can be viewed as a classifier in Y X , and vice versa.

For any distribution P over X ˆ Y , the error rate of a k -menu µ under P is defined as er P p µ q : ' P ptp x, y q P X ˆ Y : y R µ p x quq which agrees with the definition of the error rate of classifiers when the size of the menu is 1.

Definition 2.2 ( k -list learner) . A list learner A of size k P N is an algorithm which given a sequence s P Y 8 n ' 0 p X ˆ Y q n and a concept class H Ď Y X , outputs a k -menu A p s , H q . A 1 -list learner can be viewed as a multiclass learner, and vice versa.

Similar to multiclass learners, the expected error rate of a list learner A is defined as

for any concept class H P Y X and distribution P over X ˆ Y . Restricting to realizable distributions, we can define

Next, we define list PAC learning.

Definition 2.3 (List PAC learning, Charikar and Pabbaraju 2023) . For any concept class H Ď Y X and k P N , the (PAC) sample complexity M A , H : p 0 , 1 q 2 Ñ N of a k -list learner A is a mapping from p ε, δ q P p 0 , 1 q 2 to the smallest positive integer such that for every m ě M A , H p ε, δ q and every distribution P P RE p H q , P S ' P n p er P p A p S, H qq ą ε q ď δ , and we define M A , H p ε, δ q ' 8 if no such integer exists. We say H is k -list PAC learnable by A if M A , H p ε, δ q ă 8 for all p ε, δ q P p 0 , 1 q 2 . The k -list (PAC) sample complexity of H is defined as M k H p ε, δ q : ' inf k -list learner A M A , H p ε, δ q for any p ε, δ q P p 0 , 1 q 2 .

Note that H is PAC learnable by a learner A if and only if H is 1 -list learnable by A , and the PAC sample complexity of H is M H ' M 1 H . For list PAC learning, it was proved by Charikar and Pabbaraju [2023] that a concept class H is k -list learnabale if and only if d k : ' dim k p H q ă 8 , and there exists a k -list learner A k which for any P P RE p H q , δ P p 0 , 1 q , n P N , and S ' P n , satisfies that with probability at least 1 ´ δ ,

For the expected error rate, the lower bound ε k H p n q ' Ω p d k {p kn qq has been proved in Charikar and Pabbaraju [2023, Theorem 6]. However, a lower bound of the same order on the k -list PAC sample complexity is still missing in the literature. To establish a lower bound, we also need to rule out trivial classes in list learning. In analogy to Definition 1.8, we define k -nondegeneracy as follows.

Definition 2.4 ( k -nondegenerate concept class) . A concept class H P Y X is called k -nondegenerate for k P N if there exist h 1 , . . . , h k ` 1 P H and x 0 , x 1 P X such that |t h j p x 0 q : j P r k ` 1 su| ' 1 and |t h j p x 1 q : j P r k ` 1 su| ' k ` 1 . H is called k -degenerate if it is not k -nondegenerate.

We claim that only one training point is sufficient for the k -list learning of a k -degenerate concept class H . Indeed, H is k -list learnable if | H | ď k . Now, suppose that | H | ě k ` 1 and is k -degenerate. Upon observing any point p x 1 , y 1 q P X ˆ Y realizable by H , if |t h P H : h p x 1 q ' y 1 u| ď k , then x ÞÑt h p x q : h P H , h p x 1 q ' y 1 u is a k -menu which always contains the correct label. If |t h P H : h p x 1 q ' y 1 u| ě k ` 1 , then, for any x P X zt x 1 u , we must have |t h p x q : h P H , h p x 1 q ' y 1 u| ď k

because otherwise H is k -nondegenerate. Then, x ÞÑt h p x q : h P H , h p x 1 q ' y 1 u is a k -list which always contains the correct label.

Now, we are ready to present the following lower bound on the k -list PAC sample complexity. Theorem 2.5. For any k P N , k -nondegenerate concept class H Ď Y X with dim k p H q ' d k P N , ε P ´ 0 , 1 8 p k ` 1 q ¯ , and δ P ´ 0 , 1 4 p k ` 1 q ¯ , we have M k H p ε, δ q ě p d k ´ 1 q log p 2 q` 4 log p 1 { δ q 16 p k ` 1 q ε . In particular, when k ' 1 , for any ε P p 0 , 1 { 16 q and δ P p 0 , 1 { 8 q , we have

The proof of Theorem 2.5 is presented in Appendix A where we construct hard distributions based on properties of k -pseudo-cubes.

2.2 Reduction from multiclass learning to list learning

We first introduce the theorem that provides a guarantee on PAC sample complexity based on expected error rate, which will be used frequently in our analysis.

Theorem 2.6. Fix a concept class H Ď Y X and consider a learner A which satisfies ε A , H ,P p n q ď M n { n for any n P N and P P RE p H q with M n nondecreasing in the sample size n . Then, there exists a learner A 1 such that for any P P RE p H q , δ P p 0 , 1 q , n ě 4 , and the training sequence S ' P n , with probability at least 1 ´ δ , we have

The proof of Theorem 2.6 is provided in Appendix B. Now, we consider general list learners whose sizes may depend on the sample size. The theorem below states our reduction to list learning.

Theorem 2.7. Assume that there exists a list learner A list which for any H Ď Y X , D P RE p H q , n P N , and S ' D n , outputs a menu A list p S, H q of size p p H , n q satisfying ε A list , H , D ď β p H , n q{ n for some function β : ✷ Y X ˆ N Ñr 0 , 8q . Without loss of generality, we assume that p p H , n q and β p H , n q are nondecreasing in n . Then, there exist multiclass learners A red (see Algorithm 1) and A 1 red which for any concept class H of DS dimension d , D P RE p H q , δ P p 0 , 1 q , and n ě 4 , satisfy

The proof of Theorem 2.7 is presented in Appendix C. Note that the order of the error rate upper bound of the constructed multiclass learner is not smaller than that of the original list learner in the above theorem. Thus, the list learner A k of size k P N developed in Charikar and Pabbaraju [2023] cannot lead to an improved error rate of multiclass learning using our current result. The construction of A red from A list is shown in Algorithm 1.

Algorithm 1: Multiclass learner A using a list learner A

In step 6 of Algorithm 1, A G is a multiclass learner defined in Proposition H.5 in the appendix. Moreover, we prove in Proposition H.5 that for any D P RE p H q , n P N , δ P p 0 , 1 q , and S ' D n , with probability at least 1 ´ δ , we have

where dim G p H q is the graph dimension [Natarajan and Tadepalli, 1988] of H (see Definition H.1 in the appendix). The above bound for classes of finite graph dimensions is also novel in the literature.

We briefly comment on the analysis of Algorithm 1. We first apply the list learner to the first third of the training samples to obtain the menu p µ . Then, we count the number of errors ( N ) made by p µ in the last two thirds samples. Then, we consider H x 1 which is a subset of H | x 1 such that the number of errors on x 1 is bounded by N ` 1 . We show in Lemma H.9 that dim G p H x 1 q is well controlled. However, as we do not observe the label of the test point, we can only consider resampling from elements in S 2 as the new training sequence fed to A G together with the concept class H x 1 . Thus, there still exist great challenges of upper bounding the error probability for the test point that will never be sampled. We need to emphasize that the standard leave-one-out argument [Brukhim et al., 2022, Fact 14] cannot be directly applied as the definition of N that determines H x 1 only depends on S 2 but not the test point p X n ` 1 , Y n ` 1 q . We tackle this challenge by proving that some permutation of the error event together with the constraint on correctness of p µ on the last two points in x 1 when leaving the last element (i.e., the test point) out is a subset of the error event when leaving the previous element (i.e., the point in S 2 ) out. The details of the proof is presented in Appendix C.

2.3 Sampled boosting of list learners

We now build a list learner whose invocation to Theorem 2.7 yields the upper bound in Theorem 1.9. Brukhim et al. [2022, Lemma 39] proposed a list sample compression scheme of size r ' O p d 3 { 2 log p n qq for concept classes of DS dimension d and sample size n . One can show that its error rate is O pp r log p n { r q ` log p 1 { δ qq{ n q using standard techniques for sample compression schemes [David et al., 2016], which however brings the extra log factor log p n { r q . Recently, da Cunha et al. [2024] proposed stable randomized sample compression schemes for binary classification whose generalization does not induce the extra log factor in n and used this framework to analyze a subsampling-based boosting algorithm for weak learners. Motivated by its success, we extend their boosting algorithm [da Cunha et al., 2024, Algorithm 1] for multilclass list learners in Algorithm 2. Before presenting the algorithm, we first need to define the majority vote of menus. For K P N menus µ 1 , . . . , µ K each of size p , we define their majority vote to be µ ' Maj p µ 1 , . . . , µ k q with

Note that µ has size 2 p ´ 1 . For p ' 1 , the above definition recovers the majority vote of classifiers.

Algorithm 2: Sampled boosting A boost of a list learner A list

Here, γ and ν are fixed constants, enabling us to invoke weak list learners (of constant error and confidence levels) to Algorithm 2. Next, we upper bound the error rate of the boosted list learner.

Theorem 2.8. Assume that A list is a list learner with M A list , H p 1 { 2 ´ γ, ν q ă 8 for some γ P p 0 , 1 { 2 q and ν P p 0 , γ { 18 s . Then, for any D P RE p H q , n P N , and δ ą 0 , sampling S ' D n , with probability at least 1 ´ δ , the menu µ produced by A boost using A list in Algorithm 2 satisfies that

The proof of Theorem 2.8 is a generalization of the proof of da Cunha et al. [2024, Theorem 1.1] and is presented in Appendix D together with the proofs of other results in this section. Since multiclass learners are list learners of size 1, we can also boost multiclass learners using Algorithm 2. For instance, invoking the multiclass learner in Brukhim et al. [2022, Theorem 1] to Algorithm 2 and applying Theorem 2.6, we achieve the following sample complexity in multiclass learning.

There is an extra log p d q factor in the above upper bound compared to that in Theorem 1.9, which explains the reason of routing through list learning with our reduction in Theorem 2.7: the list sample compression scheme in Brukhim et al. [2022] saves a log p d q factor compared to their sample compression scheme. Therefore, we invoke their list sample compression scheme as A list to Algorithm 2 and build a list learner whose error rate depends on only one log factor in both n and d .

Theorem 2.10. There exists a list learner A L which for any H Ď Y X with dim p H q ' d and sample size n P N outputs a menu of size O ` p e ? d q ? d log p n q ˘ with ε A L , H p n q ' O ` d 3 { 2 log p d q log p n q n ˘ .

2.4 Improved upper bounds on sample complexity

Applying the list learner A L in Theorem 2.10 to our reduction, Theorem 2.7 immediately implies the following result.

Theorem 2.11. There exists a multiclass learner A multi such that for any H Ď Y X of DS dimension d , D P RE p H q , δ P p 0 , 1 q , n ě d ` 1 , and S ' D n , with probability at least 1 ´ δ , we have

which implies that

Furthermore, if there exists a list learner A goodlist of size f 1 p d q and expected error rate ε A goodlist , H p n q ď f 2 p d q{ n for some functions f 1 : N Ñ N and f 2 : N Ñ r 0 , 8q , then, there exists a multiclass learner A lin such that

The proof of Theorem 2.11 follows directly from Theorem 2.7 and Theorem 2.10 and is provided in Appendix E. Moreover, observing that dim k p H q ě dim k 1 p H q for k ă k 1 , our requirement on A goodlist does not violate the lower bound in Charikar and Pabbaraju [2023, Theorem 6].

Compared to the upper bound (1) by Brukhim et al. [2022], (7) improves the dependence of the error rate on the sample size n from O ` log p log p n qq log 2 p n q{ n ˘ to O p log p n q{ n q , which steps further towards the goal of O p 1 { n q expected error rate (treating the DS dimension as a constant). Combining (5) and (8), we arrive at (3) where the gap has been improved to the factor ? d log p d q log p d { ε q . However, we are not aware of any existing list learner satisfying the requirements of A goodlist in Theorem 2.11. Thus, we leave the construction of A goodlist as an open question.

Open Question 1. Does there exist a list learner such that given a concept class H Ď Y X , its size is f 1 p dim p H qq and its expected error rate is ε A list , H p n q ' f 2 p dim p H qq{ n for some functions f 1 : N Ñ N and f 2 : N Ñr 0 , 8q ?

Ideally, we would expect a list learner with size O p dim p H qq and expected error rate O p dim p H q{ n q as it immediately implies an upper bound M H p ε, δ q ' O pp dim p H q log p dim p H qq ` log p 1 { δ qq{ ε q which matches the lower bound in (5) up to the factor log p dim p H qq .

3 Density, DS dimension, and pivot shifting

We now introduce an alternative route toward proving the optimal sample complexity of multiclass PAC learning: bounding the density µ H : N Ñr 0 , 8q (Definition 1.7) of concept classes H . The

following proposition summarizes existing results that illustrate the role of density in multiclass learning.

Proposition 3.1 (Daniely and Shalev-Shwartz 2014, Charikar and Pabbaraju 2023, Aden-Ali et al. 2023) . For any H Ď Y X and n P N , we have

Assume that µ H p n q ď f p dim p H qq for some function f : N Ñ r 0 , 8q and all n P N . Then, there exists a learner A based on orienting the one-clusion graph (Definition 1.5) of the projected concept class (see Aden-Ali et al. [2023, Appendix A] for the formal definition of the algorithm) with sample complexity M A , H p ε, δ q ' O ` f p dim p H qq` log p 1 { δ q ε ˘ for all ε, δ P p 0 , 1 q . 1

In Proposition 3.1, the first inequality of (10) follows from Charikar and Pabbaraju [2023, Theorem 6], the last inequality of (10) follows from Daniely and Shalev-Shwartz [2014, Theorem 2], and the last paragraph follows from Aden-Ali et al. [2023, Theorem 2.2]. Thus, for sharper multiclass sample complexity, it suffices to bound the density of a concept class with some functions of its DS dimension. Furthermore, by Definition 1.7 and (2), it suffices to bound the average out-degree (Definition 1.6) of finite one-inclusion graphs. In fact, it has been conjectured that µ H p n q ď c ¨ dim p H q for some constant c ą 0 [Daniely and Shalev-Shwartz, 2014] and the question remained open since then. A positive resolution of this conjecture would immediately imply that the M H p ε, δ q ' Θ pp dim p H q ` log p 1 { δ qq{ n q by Proposition 3.1 and Theorem 2.5. It is worth mentioning that for H Ď t 0 , 1 u X , Haussler et al. [1994] proved that µ H ď 2 dim p H q (for binary classes, the DS dimension is the VC dimension), which also motivates the above conjecture. In this paper, we confirm the above conjecture for concept classes of DS dimension 1.

Theorem 3.2. For any H Ď Y X with dim p H q ' 1 , we have µ H p n q ď 2 , @ n P N . Thus, M H p ε, δ q ' Θ p log p 1 { δ q{ ε q for any positive ε, δ P O p 1 q and any H with dim p H q ' 1 .

The above theorem follows from the following fact we prove for one-inclusion graphs of DS dimension 1 concept classes. The proofs of Theorem 3.2 and Proposition 3.3 are presented in Appendix F.

Proposition 3.3. For any n P N and V n Ď Y n with | V n | ă 8 and dim p V n q ' 1 , there exists no cycle (see Definition 3.4) in the one-inclusion graph G p V n q (see Definition 1.5).

Definition 3.4 (Cycle in finite hypergraph) . A cycle of length m P N zt 1 u in a finite hyergraph G ' p V, E q consists of pairwise different vertices v 0 , . . . , v m ´ 1 P V and pairwise different edges e 0 , . . . , e m ´ 1 P E such that v j , v p j ` 1 q mod m P e j for all 0 ď j ď m ´ 1 .

We prove Proposition 3.3 by contradiction and analyzing different cases of the cycle. However, it is hard to extend such result to classes of higher DS dimensions. For general concept classes, motivated by the proof for binary classes [Haussler et al., 1994, Lemma 2.4], we also consider proving by induction on the size of the sequence the class projects to. Though the analysis for binary classes does not apply to general concept classes, we discover that the analysis in the induction step proceeds seamlessly for some special concept classes where a common label which we call a 'pivot' exists for each edge in the last dimension of size greater than 1 in its one-inclusion graph. Before summarizing this result in Lemma 3.6 below, we first introduce the definition of a 'pivot' formally.

Definition 3.5 (Pivot of finite concept class) . For any n P N zt 1 u and V n Ď Y n , we define

Then, a P Y is said to be a pivot of V n if p y 1 , . . . , y n ´ 1 , a q P V n for any p y 1 , . . . , y n ´ 1 q P P p V n q . We emphasize that when P p V n q ' H , every a P Y is a pivot of V n .

Then, we can present Lemma 3.6 whose proof is provided in Appendix G.

Lemma 3.6. Assume that for some n P N zt 1 u , any d P N , any m P r n ´ 1 s , and any H Ď Y m with dim p H q ď d and | H | ă 8 , we have avgoutdeg p G p H qq ď d . Consider an arbitrary set V n Ď Y n such that | V n | ă 8 and dim p V n q ď d . If V n has a pivot, then we have avgoutdeg p G p V n qq ď d .

Though it only works for special classes, Lemma 3.6 can serve as a building block in the induction step for the proof of avgoutdeg p G p H qq ď dim p H q for finite H Ď Y n P N Y n . Moreover, the base case n ' d ` 1 has been verified in Brukhim et al. [2022, Lemma 13]. Consequently, it suffices to extend the induction step for concept classes without a pivot. With Lemma 3.6, it is natural to consider modifying the concept class to create a pivot for it while at the same time preserving the DS dimension of the modified class nonincreasing. The technique used here is similar to shifting [Haussler, 1995, Brukhim et al., 2022], though we do not shift the whole edge 'downwards' but only shift the last label in some vertex of the edge to a candidate pivot. The difference is necessary, as it has already been shown that the DS dimension of a concept class can increase after the standard shifting [Brukhim et al., 2022, Example 19]. Thus, we name the technique used here 'pivot shifting'.

Definition 3.7 (Pivot shifting) . For any n P N zt 1 u , a P Y , and V n Ď Y n with | V n | ă 8 , we define

For any y ' p y 1 , . . . , y n ´ 1 q P P a p V n q and the edge p e n, y , n q in G p V n q , we define the set

A mapping γ : P a p V n q Ñ Y is called a pivot shifting on V n to a if γ p y q P L y for all y P P a p V n q . Let Γ a,V n denote the set of all pivot shifting on V n to a . For any γ P Γ a,V n , we define

i.e., V n,γ is obtained by replacing the label γ p y q in p y , γ p y qq with a for all y P P a p V n q .

We prove that the average out-degree does not decrease after pivot shiftings in the following lemma.

Lemma 3.8. For any a P Y , V Ď Y 8 n ' 2 Y n with | V | ă 8 , and γ P Γ a,V , we have

The proof is presented in Appendix G. A key observation for the proof is that by definition, only edges of sizes greater than one contribute to the average out-degree. However, we are not able to show that the DS dimension does not increase after some pivot shifting, which we leave as an open question. Thus, whether pivot shifting is applicable to upper bounding density with DS dimension remains open.

Open Question 2. For any d P N and any V Ď Y 8 n ' d ` 2 Y n with | V | ă 8 and dim p V q ' d , are there some a P Y and γ P Γ a,V such that dim p V γ q ď d ?

Nevertheless, we have taken a further and specific step toward the verification of the conjecture that µ H ď 2 dim p H q : a positive resolution of the above question would lead to the conclusion that µ H ď 2 dim p H q by Lemma 3.6, Lemma 3.8, and Brukhim et al. [2022, Lemma 13].

Acknowledgments and Disclosure of Funding

Shay Moran is a Robert J. Shillman Fellow; he acknowledges support by ISF grant 1225/20, by BSF grant 2018385, by Israel PBC-VATAT, by the Technion Center for Machine Learning and Intelligent Systems (MLIS), and by the the European Union (ERC, GENERALIZATION, 101039692). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.

References

Aden-Ali, Y. Cherapanamjeri, A. Shetty, and N. Zhivotovskiy. Optimal pac bounds without uniform convergence. In 2023 IEEE 64th Annual Symposium on Foundations of Computer Science (FOCS) , pages 1203-1223, Los Alamitos, CA, USA, nov 2023. IEEE Computer Society. doi: 10.1109/FOCS57990.2023.00071. URL https://doi.ieeecomputersociety.org/10. 1109/FOCS57990.2023.00071 .

Ben-David, N. Cesa-Bianchi, D. Haussler, and P. Long. Characterizations of learnability for classes of t 0 , . . . , n u -valued functions. Journal of Computer and System Sciences , 50:74-86, 1995.

Bendavid, N. Cesabianchi, D. Haussler, and P.M. Long. Characterizations of learnability for classes of {0, ..., n}-valued functions. Journal of Computer and System Sciences , 50(1):74-86, 1995. ISSN 0022-0000. doi: https://doi.org/10.1006/jcss.1995.1008. URL https://www.sciencedirect. com/science/article/pii/S0022000085710082 .

Nataly Brukhim, Daniel Carmon, Irit Dinur, Shay Moran, and Amir Yehudayoff. A characterization of multiclass learnability. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS) , pages 943-955. IEEE, 2022.

Moses Charikar and Chirag Pabbaraju. A characterization of list learnability. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing , STOC 2023, page 1713-1726, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450399135. doi: 10.1145/3564246.3585190. URL https://doi.org/10.1145/3564246.3585190 .

Arthur da Cunha, Kasper Green Larsen, and Martin Ritzert. Boosting, voting classifiers and randomized sample compression schemes. arXiv preprint arXiv:2402.02976 , 2024.

Amit Daniely and Shai Shalev-Shwartz. Optimal learners for multiclass problems. In Conference on Learning Theory , pages 287-316. PMLR, 2014.

Ofir David, Shay Moran, and Amir Yehudayoff. Supervised learning through the lens of compression. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper_files/paper/2016/file/ 59f51fd6937412b7e56ded1ea2470c25-Paper.pdf .

Hanneke. The optimal sample complexity of PAC learning. Journal of Machine Learning Research , 17(38):1-15, 2016.

Steve Hanneke, Shay Moran, and Qian Zhang. Universal rates for multiclass learning. In The Thirty Sixth Annual Conference on Learning Theory , pages 5615-5681. PMLR, 2023.

Haussler. Sphere packing numbers for subsets of the Boolean n-cube with bounded VapnikChervonenkis dimension. Journal of Combinatorial Theory A , 69(2):217-232, 1995.

Haussler, N. Littlestone, and M. Warmuth. Predicting t 0 , 1 u -functions on randomly drawn points. Information and Computation , 115(2):248-292, 1994.

Kamran Kowsari, Kiana Jafari Meimandi, Mojtaba Heidarysafa, Sanjana Mendu, Laura Barnes, and Donald Brown. Text classification algorithms: A survey. Information , 10(4):150, 2019.

Tao Li, Chengliang Zhang, and Mitsunori Ogihara. A comparative study of feature selection and multiclass classification methods for tissue classification based on gene expression. Bioinformatics , 20(15):2429-2437, 2004.

K. Natarajan. On learning sets and functions. Machine Learning , 4:67-97, 1989.

Balas K Natarajan and Prasad Tadepalli. Two new frameworks for learning. In Machine Learning Proceedings 1988 , pages 402-415. Elsevier, 1988.

Waseem Rawat and Zenghui Wang. Deep convolutional neural networks for image classification: A comprehensive review. Neural Computation , 29(9):2352-2449, 2017. doi: 10.1162/neco_a_00990.

G. Valiant. A theory of the learnable. Communications of the ACM , 27(11):1134-1142, November 1984.

Vapnik and A. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Proc. USSR Acad. Sci. , 181(4):781-783, 1968.

A Lower bound

Before proving Theorem 2.5, we first introduce two lemmas regarding k -pseudo-cubes that will be used in the proof.

Lemma A.1. For any positive integers k, d , any label class Y , any k -pseudo-cube H Ď Y d of dimension d , any j P r d s , and any label y P Y , define H j y : ' t h P H : h p j q ' y u . Then, we have

Proof. When d ' 1 , the result follows trivially from the definition of k -pseudo-cubes. We prove the result for d ě 2 by contradiction. Suppose on the contrary that there exist some j P r d s and y P Y such that | H j y | ą | H | k ` 1 . The definition of pseudo-cubes implies that | H | ě k ` 1 . Then, there exist h, h 1 P H j y with h ‰ h 1 . Let t f 1 , . . . , f k u and t f 1 1 , . . . , f 1 k u denote the set of j -neighbors of h and h 1 in H respectively. Since h ‰ h 1 and h p j q ' y ' h 1 p j q , there exists some j 1 P r d szt j u such that h p j 1 q ‰ h 1 p j 1 q . It follows that f i p j 1 q ' h p j 1 q ‰ h 1 p j 1 q ' f 1 l p j 1 q and thus f i ‰ f 1 l for any i, l P r k s . Then, we have

which is a contradiction. Thus, we must have | H j y | ď | H | k ` 1 .

Lemma A.2. For any integer k ě 1 , d ě 2 , n P r d ´ 1 s , and 1 ď j 1 ă ¨ ¨ ¨ ă j n ď d , any label class Y , any k -pseudo-cube H Ď Y d of dimension d , and any hypothesis g P H , define J : ' p j 1 , . . . , j n q and K ' p k 1 , . . . , k d ´ n q such that 1 ď k 1 ă ¨ ¨ ¨ ă k d ´ n ď d and t j 1 , . . . , j n , k 1 , . . . , k d ´ n u ' r d s . Then, H g, J : ' t h | K : h P H,h p j i q ' g p j i q , @ i P r n su is a k -pseudo-cube of dimension d ´ n .

Proof. For any f P H g,J , there exists some h P H such that f ' h | K . Then, for any i P r d ´ n s , there exists k distinct h 1 , . . . , h k P H such that h m p k i q ‰ h p k i q and h m p l q ' h p l q for all l P r d szt k i u and m P r k s . Since k i R t j 1 , . . . , j n u , we have h | J ' h m | J ' g | J and thus f m : ' h m | K P H g, J for all m P r k s . Then, we have f p i q ' h p k i q ‰ h m p k i q ' f m p i q and f p l q ' h p k l q ' h m p k l q ' f m p l q for any l P r d ´ n szt i u and m P r k s , which implies that H g, J is a k -pseudo-cube of dimension d ´ n .

Proof of Theorem 2.5. Consider an arbitrary k -nondegenerate concept class H Ď Y X for some k P N . Let x 0 , x 1 P X and h 1 , . . . , h k ` 1 P H witness the k -nondegeneracy as specified in Definition 2.4. For any ε P p 0 , 1 { 2 p k ` 1 qq and δ P p 0 , 1 q , let I ' Bernoulli pp k ` 1 q ε q and J ' Unif pr k ` 1 sq . Then, for any j P r k ` 1 s , p x I , h j p x I qq follows the H -realizable distribution P ε,j over X ˆ Y defined by P ε,j ptp x 0 , h j p x 0 qquq ' 1 ´p k ` 1 q ε and P ε,j ptp x 1 , h j p x 1 qquq ' p k ` 1 q ε . Sample p I 1 , . . . , I n q ' Bernoulli p ε q n independent of p I, J q and define S ' pp x I 1 , h J p x I 1 q , . . . , p x I n , h J p x I n qq and S 1 ' pp x 0 , h J p x 0 q , . . . , p x 0 , h J p x 0 qq . Then, for any k -list learner A , we have

where the last inequality follows from the facts that h J p x 0 q ' h 1 p x 0 q , |t h 1 p x 1 q , . . . , h k ` 1 p x 1 qu| ' k ` 1 , h J p x 1 q ' Unif pt h 1 p x 1 q , . . . , h k ` 1 p x 1 quq . Since J ' Unif pr k ` 1 sq , there exists j P r k ` 1 s such that

and

where S j : ' pp x I 1 , h j p x I 1 qq , . . . , p x I n , h j p x I n qqq ' P n ε,j is independent of p x I , h j p x I qq ' P ε,j . Since 1 2 p k ` 1 q ε ď 1 ´p k ` 1 q ε p k ` 1 q ε ă 1 ´ log p 1 ´p k ` 1 q ε q for ε P p 0 , 1 { 2 p k ` 1 qq , if n ď log p 1 { δ q 2 p k ` 1 q ε ă log p δ q log p 1 ´p k ` 1 q ε q , we have

Then, with probability greater than δ , we have er P ε,j p A p S j , H qq ě ε , which implies that

Next, consider an arbitrary concept class H Ď Y X such that dim k p H q ' d k P N zt 1 u for some k P N . Then, there exist a sequence x : ' p x 1 , . . . , x d k q P X d k and a k -pseudo-cube H Ď H | x of dimension d k . Denote the elements in H with y 1 ' p y 1 1 , . . . , y 1 d k q , . . . , y | H | ' p y | H | 1 , . . . , y | H | d k q . For any ε P p 0 , 1 { 8 p k ` 1 qq , consider the categorical distribution Q ε over r d k s defined by Q ε pt 1 uq ' 1 ´ 4 p k ` 1 q ε and Q ε pt i uq ' 4 p k ` 1 q ε d k ´ 1 for i P r d k szt 1 u . Let J ' Unif pr| H |sq . For any i P r d k szt 1 u and I ' p i 1 , . . . , i m q P pr d k szt i uq m with m P r d k ´ 1 s and i 1 ă ¨ ¨ ¨ ă i m , define I 1 : ' p i 1 1 , . . . , i 1 d k ´ m q such that i 1 1 ă ¨ ¨ ¨ ă i 1 d k ´ m and t i 1 , . . . , i m , i 1 1 , . . . , i 1 d k ´ m u ' r d k s . Then, we have that conditional on p i 1 , y J i 1 q , . . . , p i m , y J i m q , y J | I 1 follows the uniform distribution over the set H y J , I which is a k -pseudo-cube by Lemma A.2. Consequently, we can apply Lemma A.1 to conclude that

for any y P Y and i 1 P r d k ´ m s , which immediately implies that

for any distinct v 1 , . . . , v k P Y .

Let p I, I 1 , . . . , I n q ' Q n ` 1 ε for n P N be independent of J . Define S : ' pp x I 1 , y J I 1 q , . . . , p x I n , y J I n qq . For any k -list leaner A , by (11), we have

Since J ' Unif pr| H |sq , there exists j ' j p A , H q P r| H |s such that

where S j : ' pp x I 1 , y j I 1 q , . . . , p x I n , y j I n qq . Note that if we define the distribution P ε,j over X ˆ Y by

then, we have p S, p x I , y j I qq ' P n ` 1 ε,j . For any n ď p d k ´ 1 q log p 2 q 8 p k ` 1 q ε ă log p 1 { 2 q log p 1 ´ 4 p k ` 1 q ε {p d k ´ 1 qq , we have ´ 1 ´ 4 p k ` 1 q ε d k ´ 1 ¯ n ą 1 2 and

Now, we define the algorithm A 1 by

for any s P Y 8 m ' 0 p X ˆ Y q m . Then, we have

On the other hand, the definition of A 1 yields that y j 1 P A 1 p S j , p x 1 qq and hence

Suppose the following holds and therefore

which contradicts (12). Thus, we can conclude that (13) is false, i.e.,

for n ď p d k ´ 1 q log p 2 q 8 p k ` 1 q ε . For δ P p 0 , 1 { 4 p k ` 1 qs , we have P p er P ε,j p A p S j , H qq ą ε q ą δ and thus

In conclusion, for ε, δ ' O p 1 { k q , we have

B Proof of Theorem 2.6

Proof. Consider an algorithm A : pY 8 k ' 1 p X ˆ Y q k q ˆ ✷ Y X Ñ Y X which for any hypothesis class H Ď Y X , H -realizable distribution D , and n P N , satisfies that

where M n is nondecreasing in the sample size n .

Since er P ε,j p A 1 p S j , H qq ď er P ε,j p A p S j , H qq , the above inequality implies that

For an arbitrary sequence S ' pp x 1 , y 1 q , . . . , p x n , y n q , p x n ` 1 , y n ` 1 qq P p X ˆ Y q n ` 1 , define S ´ i to be the subsequence of S excluding p x i , y i q for any i P r n ` 1 s . Let p T m , p X,Y qq ' Unif p S q m ` 1 where m P N , T m P p X ˆ Y q m and p X,Y q P X ˆ Y . Moreover, for any i P r n ` 1 s , let T i m ' Unif p S ´ i q m . Then, for any algorithm A : Y 8 k ' 1 p X ˆ Y q k ˆ ✷ Y X Ñ Y X and concept class H Ď Y X , we have

Note that if S is consistent with H , then Unif p S q is H -realizable.

Next, define the algorithm A Maj ,m : pY 8 k ' 1 p X ˆ Y q k q ˆ Y X Ñ Y X by

By the definition above, for any permutation R π of R P p X ˆ Y q n , we have

Moreover, for any i P r n ` 1 s , the above definition yields that

Then, by (15), we have

Choosing m ' t p n ` 1 q{ 2 u and defining A Maj : ' A Maj , t p n ` 1 q{ 2 u , for any H -realizable sequence S , by (14) and the above results, we have

for any n ě 4 . (16) and (17) imply that the algorithm A Maj satisfies Assumptions 2.1 and 2.3 in Aden-Ali et al. [2023].

Now, we can define the randomized algorithm A Ran which given a sample S P p X ˆ Y q n with n P N and a concept class H , outputs the classifier A Maj p S, H q if n ď 3 and outputs a random classifier following the uniform distribution over the sequence p A Maj p S ď t , H qq t n { 4 u ď t ď 4 t n { 4 u ´ 1 if n ě 4 , i.e.,

where S ď t denotes the subsequence of S consisting of the first t elements in S . Then, by Aden-Ali et al. [2023, Theorem 2.1], for any n ě 4 , H -realizable distribution D , and confidence parameter

δ P p 0 , 1 q , given a training sample S ' D n , we have

with probability at least 1 ´ δ .

C Proof of Theorem 2.7

The following proof requires several lemmas from Appendix H.

Proof of Theorem 2.7. Assume that we have access to a list learning algorithm A list which for any concept class H Ď Y X of DS dimension d , any H -realizable distribution D , any n P N , and p S, p X,Y qq ' D n ` 1 , outputs a menu A list p S, H q of size p p n q P N satisfying

for some function β : Y X ˆ N Ñr 0 , 8q .

✷ For n 1 , n 2 P N , let p S 1 , S 2 , p X,Y qq ' D n 1 ` 2 n 2 ` 1 where S 1 P p X ˆ Y q n 1 , S 2 P p X ˆ Y q 2 n 2 , and p X,Y q P X ˆ Y . Let p µ ' A list p S 1 , H q . According to the property of A list , the size of p µ is p p n 1 q and we have

For notational convenience, define S 1 : ' p S 2 , p X,Y qq , S 2 : ' p S 1 , S 2 , p X,Y qq , and enumerate the elements of S 1 as

where p X 2 n 2 ` 1 , Y 2 n 2 ` 1 q denotes p X,Y q . We now define and

It follows from the property of p µ and Lemma H.9 that

and, conditional on S 1 ,

Sample p I , I q ' Unif pr 2 n 2 ` 1 sq n 2 ` 1 independent of S 2 where I ' p I 1 , . . . , I n 2 q P r 2 n 2 ` 1 s n 2 and I P r 2 n 2 ` 1 s . Then, we define the sequence T p S 1 , I q : ' pp I 1 , Y I 1 q , . . . , p I n 2 , Y I n 2 qq and the classifier

for any i P r 2 n 2 ` 1 s , where A G is the algorithm specified in Proposition H.5. Since conditional on S 2 , the distribution of each element in T p S 1 , I q is H S 1 -realizable, by Corollary H.6, there exists some constant C 1 ą 0 such that

for some constant C 1 , C 2 , C 3 ą 0 .

For I 1 ' p I 1 1 , . . . , I 1 n 2 q ' Unif pr 2 n 2 sq n 2 independent of all other random variables, define T i p S 1 , I 1 q : ' pp ρ i p I 1 1 q , Y ρ i p I 1 1 q q , . . . , p ρ i p I 1 n 2 q , Y ρ i p I 1 n 2 q qq where ρ i : r 2 n 2 s Ñ r 2 n 2 ` 1 szt i u , k ÞÑ k ✶ k ă i `p k ` 1 q ✶ k ě i for all i P r 2 n 2 ` 1 s . Consider the classifier r h S 1 , I 1 ,i : ' A G p T i p S 1 , I 1 q , H S 1 q for each i P r 2 n 2 ` 1 s . Since Y i is not used in the construction of r h S 1 , I 1 ,i , we can also denote r h S 1 , I 1 ,i as r h S 1 ´ i ,X i , I 1 ,i , where S 1 ´ i : ' pp X 1 , Y 1 q , . . . , p X i ´ 1 , Y i ´ 1 q , p X i ` 1 , Y i ` 1 q , . . . , p X 2 n 2 ` 1 , Y 2 n 2 ` 1 qq . Thus, treating S : ' p S 1 , S 2 q ' D n 1 ` 2 n 2 as the training sample, we can define the classifier p h S P Y X by

where we emphasize that the RHS depends on S 1 through the construction of p µ . Then, recalling that p X,Y q ' D is independent of S , our task is to upper bound the expectated error rate of p h S :

We first relate p h S 1 , I to r h S 1 , I 1 ,i for i P r 2 n 2 ` 1 s :

which implies that

Conditional on I 1 and S 1 , for any i P r 2 n 2 ` 1 s and any sequence s P p X ˆ Y q 2 n 2 ` 1 , we let r h s , I 1 ,i denote the classifier when replacing S 1 with s in r h S 1 , I 1 ,i , i.e., r h s , I 1 ,i ' A G p T i p s , I 1 q , H s q . For any i P r 2 n 2 ` 1 s , define the set

and the permutation

We also define the set

We would like to show that π i p B q Ď B i for all i P r 2 n 2 s . For any s ' pp x 1 , y 1 q , . . . , p x 2 n 2 ` 1 , y 2 n 2 ` 1 qq P B , we let p x i j , y i j q P X ˆ Y denote the j -th element of π i p s q for each j P r 2 n 2 ` 1 s . By the definition of π i and T i , we have

Define

Since y 2 n 2 P p µ p x 2 n 2 q and y 2 n 2 ` 1 P p µ p x 2 n 2 ` 1 q , we have and therefore,

where for any h P Y 2 n 2 ` 1 , κ i p h q P Y 2 n 2 ` 1 is defined by

κ i p h qp j q : ' h p ρ ´ 1 i p j qq ✶ j ‰ i ` h p 2 n 2 ` 1 q ✶ j ' i , @ j P r 2 n 2 ` 1 s , i.e., h p j q ' κ i p h qp ρ i p i qq for j P r 2 n 2 s and h p 2 n 2 ` 1 q ' κ i p h qp i q . Note that κ i is a bijection from H s to H π i p s q . Thus, for any h P H π i p s q , we have

Similarly, for any h P H s , we have

Given the above analysis, we have

which immediately implies that

and thus π i p s q P B i . Since i P r 2 n 2 s and s P B is arbitrary, we have π i p B q Ď B i for any i P r 2 n 2 s . Then, conditional on I 1 and S 1 , we have

Since B Ď B 2 n 2 ` 1 holds trivially, we have

which leads to

Now, for any n P N such that n ě 3 , setting n 1 ' n ´ 2 t n { 3 u , n 2 ' t n { 3 u , p S, p X,Y qq ' D n ` 1 with S P p X ˆ Y q n and p X,Y q P X ˆ Y , we have

Then, when β p H , n q and p p n q are nondecreasing in n , by Theorem 2.6, there exists a learner A 1 such that for any n ě 4 and δ P p 0 , 1 q , with probability at least 1 ´ δ over S ' D n , we have

D Proofs of the results in Section 2.3

In this section, we provide the proofs of Theorem 2.8, Corollary 2.9, and Theorem 2.10 in Section 2.3.

Proof of Theorem 2.8. Assume that A list satisfies that M A list , H p 1 { 2 ´ γ, ν q ă 8 for some γ P p 0 , 1 { 2 q and ν P p 0 , γ { 18 s . Define the random variable J k : ' 1 er D k p µ k qą 1 { 2 ´ γ for any k P r K s . Then, we have

Define the event E : ' ! ř K k ' 1 J k ě 2 νK ) . By the multiplicative Chernoff bound, we have

as K ě 3 log p 1 { δ q ν . Define Z k : ' ř n i ' 1 D k ptp x i , y i quq exp ` ´ α ` 2 1 y i P µ k p x i q ´ 1 ˘˘ for all k P r K s . Since D K ` 1 is a probability distribution over S , we have

which implies that

For any k P r K s , we have

If J k ' 0 , we have

If J k ' 1 , we have

Then, under the event E c , we can upper bound

where the second last inequality follows from ν ď γ 18 ď γ 4 p γ ` 4 q and the last inequality follows from K ě 4 log p n { δ q γ . By (20) and the above inequality, we have that on E c ,

which implies that exp ´ ´ α ř K k ' 1 ` 2 1 y i P µ k p x i q ´ 1 ˘ ¯ ď δ for all i P r n s . It follows that

for f p x, y q : ' 1 K ř K k ' 1 ` 2 1 y P µ k p x q ´ 1 ˘ , @p x, y q P X ˆ Y . Thus, we have

Since by definition,

we can conclude that on E c ,

i.e., with probability at least 1 ´ δ , y i P µ p x i q for all i P r n s . Moreover, by da Cunha et al. [2024, Lemma 3.3], the randomized compression scheme S ÞÑ p S 1 , . . . , S K q is stable. Thus, we can apply da Cunha et al. [2024, Theorem 1.2] with compression size s n ' mK ' O ´ M A list , H p 1 { 2 ´ γ,ν q log p n { δ q γ ¯ to conclude the proof.

For the proofs of Corollary 2.9 and Theorem 2.10, we will need the following Lemma.

Lemma D.1. If x ą 0 satisfies x ď a log p x { a q ` b for some a, b ą 0 , then, we have x ď 2 a ` 2 b .

Proof. Define f p x q : ' x ´ a log p x { a q ´ b for x ą 0 . Then, we have f 1 p x q ' x ´ a x , which implies that f decreases with x for x P p 0 , a q and increases with x for x ą a . Since 2 a ` 2 b ą a , it suffices to prove that f p 2 a ` 2 b q ě 0 . Indeed,

Proof of Corollary 2.9. By Brukhim et al. [2022, Theorem 36] (choosing t ' r ? d s ), there exists an n Ñ r sample compression scheme A M for any H Ď Y X of DS dimension dim p H q ' d ă 8 where

Then, by David et al. [2016, Theorem 3.1], there exists a universal constant C ą 0 such that for any D P RE p H q , δ P p 0 , 1 q , n P N large enough, and S ' D n , letting h S : ' A M p S, H q denote the output classifier of the above n Ñ r sample compression scheme A M , we have that

with probability at least 1 ´ δ . Thus, for any ε P p 0 , 1 q , if

we have er D p h S q ď ε with probability at least 1 ´ δ . By Lemma D.1, it suffices to require

Applying the upper bound of r in (21), it suffices to require

for some universal constant C 1 ą 0 . By Lemma D.1 again, it suffices to require

Since log p log p 1 { ε q{ ε q ' log p 1 { ε q ` log log p 1 { ε q ď 2 log p 1 { ε q , it suffices to require

for some universal constant C 2 ą 0 . Applying the upper bound of p in (21), it suffices to require that

for some universal constant C 3 ą 0 . For ε ' 1 { 6 and δ ' 1 { 54 , we require that

for some universal constant C 1 ą 0 . If C 1 p ? 2 C 1 ` 1 q d 3 { 2 log p d q log ` p ? 2 C 1 ` 1 q d 3 { 2 log p d q ˘ ď n ď e d ? 2 C 1 d , we have ? 2 C 1 d 3 { 2 log p d q ě d log log p n q and

Therefore, we can conclude that if n ě C 1 p ? 2 C 1 ` 1 q d 3 { 2 log p d q log ` p ? 2 C 1 ` 1 q d 3 { 2 log p d q ˘ ' Θ p d 3 { 2 log 2 p d qq , we have P p er D p h S q ą 1 { 6 q ď 1 { 54 , i.e.,

Let A B denote the multiclass learner output by A boost in Algorithm 2 using A M as the weak list learner of size 1. Then, by Theorem 2.8, we have

Next, by Theorem 2.6, there exists a multiclass learner A such that for any D P RE p H q , δ P p 0 , 1 q , n P N , and S ' D n , with probability at least 1 ´ δ , we have

Setting δ ' 1 { n and observing that er D P r 0 , 1 s , it follows that ε A , H p n q ' O ´ d 3 { 2 log 2 p d q log p n q n ¯ . Moreover, by Lemma D.1, for any ε P p 0 , 1 q , n ě O ´ d 3 { 2 log 2 p d q log p d { ε q` log p 1 { δ q ε ¯ implies that er D p A p S, H qq ď ε with probability at least 1 ´ δ . Thus, we have

Proof of Theorem 2.10. By Brukhim et al. [2022, Theorem 39] (choosing t ' r ? d s ), there exists an n Ñ r list sample compression scheme A list of size p ' O pp e ? d q ? d log p n qq for any H Ď Y X of DS dimension dim p H q ' d ă 8 where

Define the following loss function for menus µ ,

ℓ p µ, p x, y qq : ' ✶ y R µ p x q , @p x, y q P X ˆ Y . Then, we can apply the proof of David et al. [2016, Theorem 3.1] with the loss function ℓ to show that there exists a universal constant C ą 0 such that for any n Ñ r list sample compression scheme A LSC for H , any D P RE p H q , δ P p 0 , 1 q , n P N large enough, and S ' D n , letting µ S : ' A LSC p S, H q denote the output menu, we have

with probability at least 1 ´ δ . Thus, for any ε P p 0 , 1 q , if

we have er D p A LSC p S, H qq ď ε with probability at least 1 ´ δ . By Lemma D.1, it suffices to require

Since A list is an n Ñ r list sample compression scheme for H with r bounded in (22), The above results imply that in order for P p er D p A list p S, H qq ą ε q ď δ , it suffices to require

for some universal constant C 1 ą 0 . By Lemma D.1 again, it suffices to require

Since log p log p 1 { ε q{ ε q ' log p 1 { ε q ` log log p 1 { ε q ď 2 log p 1 { ε q , it suffices to require

for some universal constant C 2 ą 0 , which implies that

Now, we can define A L to be the list learner output by A boost in Algorithm 2 using A list as the weak list learner of size O pp e ? d q ? d log p n qq for training sample size n P N . Then, by Theorem 2.8, we have that for any training sample size n P N ,

and the size of A L is also O pp e ? d q d log p n qq .

E Proof of Theorem 2.11

Proof. By Theorem 2.10, there exists a list learner A L with p p H , n q ' O ` p e ? d q ? d log p n q ˘ and β p H , n q ' O ` d 3 { 2 log p d q log p n q ˘ for any n P N and concept class H Ď Y X with dim p H q ' d P N in the context of Theorem 2.7. Thus, by Theorem 2.7, there exists a multiclass learner A multi ' A 1 red such that for any D P RE p H q , any δ P p 0 , 1 q , S ' D n , and n 1 ' n ´ 2 t n { 3 u , with probability at least 1 ´ δ ,

For any ε P p 0 , 1 q , by Lemma D.1, if

then, we have i.e.,

It follows that

(9) follows directly from (6) by plugging in p p H , n q ' f 1 p d q and β p H , n q ' f 2 p d q for d ' dim p H q and any n P N .

F Classes of DS dimension 1

In this section, we present the proof of Proposition 3.3 and Theorem 3.2.

Proof of Proposition 3.3. Let E n denote the edge set of the hypergraph G p V n q . Suppose on the contrary that there exists a cycle consisting of pairwise different vertices y 0 , . . . , y m ´ 1 P V n and pairwise different edges e 0 ' p e i 0 ,f 0 , i 0 q , . . . , e m ´ 1 ' p e i m ´ 1 ,f m ´ 1 , i m ´ 1 q P E n for some m P t 4 , . . . , | V n |u such that y j , y p j ` 1 q mod m P p e i j ,f j , i j q for all 0 ď j ď m ´ 1 . Since the edges are pairwise different, by the definition of E n , we have i j ‰ i p j ´ 1 q mod m for all 0 ď j ď m ´ 1 .

Define a : ' i m ´ 1 P r n s , b : ' i 0 P r n s , p 0 : ' y 0 a ' y 1 a , p ´ 1 : ' y m ´ 1 a ‰ p 0 , and q 0 : ' y 0 b ' y m ´ 1 b . Then, we have a ‰ b .

For any k P N , we define

By definition, we have p 0 , . . . , p K are pairwise different. There are two cases depending on the values of q 1 , . . . , q K as follows.

Suppose that there exists some k P r K s such that q k ‰ q 0 . Define l 0 : ' 0 and

for all w P N with the convention that inf H'`8 . Define

We have v P r K s and q l w ‰ q l w ´ 1 for all w P r v s . Note that

Since it always holds that q K ' q l v , there are two cases depending on the value of q l v and q 0 as follows.

1.1 Suppose that q l v ‰ q 0 . Then, V n | p a,b q contains the following pseudo-cube of dimension 2:

which contradicts the assumption that dim p V n q ' 1 .

for all w P r v s , and

1.2 Suppose that q l v ' q 0 . Then, V n | p a,b q contains the following pseudo-cube of dimension 2:

which contradicts the assumption that dim p V n q ' 1 .

Thus, Case 1 does not exist.

Suppose that q k ' q 0 for all k P r K s . Since y 0 a ' p 0 ' p j 1 a , y 0 b ' q 0 ' q 1 ' p j 1 b , and y 0 ‰ y j 1 , there must exist some c P r n szt a, b u such that y m ´ 1 c ' y 1 c ' y 0 c ‰ y j 1 c ' y j 1 ` 1 c . We define r 0 : ' y 0 c and r k : ' y j k c ' y j k ` 1 c for k P r K s . There are two cases depending on the value of r k for k P r K s as follows.

2.1 Suppose that there exists some k P r K s such that r k ‰ r 1 . Similar to Case 1, we define l 0 : ' 1 and

for w P N . Define v : ' max t w P N : l w ‰ `8u . We have v P r K s and r l w ‰ r l w ´ 1 for all w P r v s . Note that

for all w P r v s , and

Since it always holds that r K ' r l v , there are two cases depending on the value of q l v and q 0 as follows.

2.1.1 Suppose that r l v ‰ r 0 . Then, V n | p a,c q contains the following pseudo-cube of dimension 2:

which contradicts the assumption that dim p V n q ' 1 .

2.1.2 Suppose that r l v ' r 0 . Then, V n | p a,c q contains the following pseudo-cube of dimension 2:

which contradicts the assumption that dim p V n q ' 1 . 2.2 Suppose that r k ' r 1 for all k P r K s . Then, we have

which implies that V n | p a,c q contains a pseudo-cube of dimension 2.

Thus, Case 2 does not exist either.

In conclusion, there exists no cycle in the hypergraph G p V n q ' p V n , E n q .

Proof of Theorem 3.2. It suffices to show that the for any n P N and V n Ď Y n , the average degree of G ' G p V n q ' p V n , E n q with | V n | ă 8 and dim p V n q ' 1 is at most 2.

We prove by induction on | V n | . When | V n | ' 1 , we have E n ' H and avgdeg p G q ' 0 ă 2 . When | V n | ' 2 , we have ř e P E n : | e |ě 2 | e | ď 2 and avgdeg p G q ď 1 ă 2 . Suppose that avgdeg p G q ď 2 for any V n of size | V n | ď m with some m P N . When | V n | ' m ` 1 , since there is no cycle in G according to Proposition 3.3, the set of vertices of degree 1 p V 1 n : ' t y P V n : deg p y q ' 1 uq is not empty. Define V 2 n : ' V n z V 1 n and E 2 n to be the edge set such that p V 2 n , E 2 n q is the one-inclusion graph on V 2 n . Then, we have | V 2 n | ' | V n | ´ | V 1 n | ď m and

because deleting a vertex of degree 1 decreases the total degree by at most 2. By the induction hypothesis, we have ř e P E 2 n | e | ď 2 | V 2 n | . Thus,

which implies that avgdeg p G q ď 2 . By induction, avgdeg p G q ď 2 for any V n with | V n | ă 8 and dim p V n q ' 1 .

G Pivot shifting

In this section, we present the proofs of Lemma 3.6 and Lemma 3.8.

Proof of Lemma 3.6. For notational convenience, we let V 1 n ´ 1 denote P p V n q which is defined in Definition 3.5. Define

and we have

For any y P Y , we define and by definition, we also have

We claim that dim p V n,y q ď d ´ 1 for all y P Y zt a u . Suppose on the contrary that dim p V n,y q ą d ´ 1 . Since dim p V n q ' d , we have dim p V n,y q ' d and there exists a set i : ' t i 1 , . . . , i d u Ď r n s with i 1 ă i 2 ă ¨ ¨ ¨ ă i d such that V n,y | i contains a pseudo-cube H y of dimension d . Since for any p y 1 , . . . , y n q P V n,y , we have y n ' y , it must hold that i d ď n ´ 1 . Now, we can define

By the assumption that | V 1 n ´ 1 | ' | V n,a | , we have | H y,a | ' 2 | H y | and H y,a Ď V n | p i 1 ,...,i d ,n q . For any k P r d s and any p y i 1 , . . . , y i d , y q , p y i 1 , . . . , y i d , a q P H y,a , since there exists an k -neighbor of p y i 1 , . . . , y i d q denoted by p y i 1 , . . . , y 1 i k , . . . , y i d q in H y , p y i 1 , . . . , y 1 i k , . . . , y i d , y 1 q P H y,a is a k -neighbor of p y i 1 , . . . , y i d , y 1 q in H y,a for y 1 ' y, a . Moreover, p y i 1 , . . . , y i d , y q is a p d ` 1 q -neighbor of p y i 1 , . . . , y i d , a q in H y,a and vise-versa. Thus, H y,a is a pseudo-cube of dimension d ` 1 , which contradicts the assumption that dim p V n q ď d . Therefore, we must have dim p V n,y q ď d ´ 1 . Then, by the induction hypothesis, we have for any y P Y zt a u ,

Let E n denote the edge set in G p V n q and E n ´ 1 denote the edge set in G p V n ´ 1 q . For any y P Y , define

By the assumption on a P Y , we have | V n | ' | V 2 n ´ 1 | ` ř y P Y | V n,y | , | V 1 n ´ 1 | ' | V n,a | , and | V n | ´ | V n ´ 1 | ' ř y P Y zt a u | V n,y | . Defining

and let E 1 n,y denote the edge set in G p V n,y q ; for any e ' p e i,f , i q P E n,y , we define

Then, we have | e | ' s 1 p e q ` s 2 p e q , E n ' pY y P Y E n,y q Y E n n , and

Note that by the induction hypothesis, we have

Summarizing the results above, we have

Proof of Lemma 3.8. Consider arbitrary n P N zt 1 u and V n Ď Y n . By the definition of P a p V n q , we have | V γ n | ' | V n | . Let E n denote the edge set of G p V n q and E n,γ denote the edge set of G p V γ n q . It suffices to prove that ř e P E n p| e | ´ 1 q ď ř e P E n,γ p| e | ´ 1 q . Define for all i P r n s . By the definition of V , we have

For any i P r n s and f : r n szt i u Ñ Y , we define

to distinguish edges in E n and E n,γ . For any y P Y , i P r n ´ 1 s , and f : r n ´ 1 szt i u Ñ Y , define f y : r n szt i u Ñ Y such that f y | r n ´ 1 szt i u ' f and f y p n q ' y . Then, we have which implies that

and from (23),

Thus, we can conclude that avgoutdeg p G p V γ n qq ě avgoutdeg p G p V n qq for any γ P Γ a,V n .

H Lemmas regarding graph dimension

In this section, we provide the technical lemmas on learning with finite graph dimension and bounding the graph dimension of certain classes. Those lemmas are used in the proof of Theorem 2.7.

H.1 Learning algorithm for classes with finite graph dimension

We first provide the definition of graph dimension.

Definition H.1 (Graph dimension) . For H Ď Y X and n P N , x ' p x 1 , . . . , x n q P X n is said to be G-shattered by H is there exists f : r n s Ñ Y such that for any i Ď r n s , there exists g P H satisfying g p x i q ' f p i q for all i P i and g p x i q ‰ f p i q for all i P r n sz i . The graph dimension of H , denoted as dim G p H q is the maximum size of a G-shattered sequence.

Define Log : r 0 , 8q Ñ r 1 , 8q , x ÞÑ log p x _ e q where x _ e ' max t x, e u . For any H Ď Y X , n P r dim G p H qs , and x ' t x 1 , . . . , x n u Ď X that is G-shattered by H , there exists f : r n s Ñ Y such that for any i Ď r n s , there exists g P H satisfying g p x i q ' f p i q for all i P i and g p x i q ‰ f p i q for all i P r n sz i . Thus, we can define

For general n P N , x Ď X with | x | ' n , and f P Y n , we define

Note that τ H p n q ' | H p x q| ' 2 n for any n P r dim G p H qs . We have the following lemma.

Lemma H.2. For any H Ď Y X with dim G p H q ' d , we have τ H p n q ď ř d i ' 0 ` n i ˘ . In particular, if n ě d , then τ H p n q ď p en { d q d .

Proof. We first prove by induction on n that for any x ' t x 1 , . . . , x n u Ď X and f P Y n ,

For n ' 1 , it is obviously that | H f p x q| ď 2 and |t x 1 Ď x : H G-shatters x 1 u| ě 1 . If |t x 1 Ď x : H G-shatters x 1 u| ' 1 , then x is not G-shattered by H , which implies that | H f p x q| ď 1 . Thus, (24) holds. Now, suppose that (24) holds for any k ă n . Consider s x : ' t x 2 , . . . , x n u ,

We have | H f p x q| ' | Y 0 | ` | Y 1 | and | Y 0 | ' | H f p s x q| . Then, by the induction hypothesis, we have

For any y P Y , define

Then, H y G-shatters x 1 Ď s x implies that H G-shatters x 1 Yt x 1 u . We also have Y 1 ' H f p 1 q f p s x q . It follows from the induction hypothesis that

In conclusion, we have

which is exactly (24). By (24), we have

For any (measurable) classifier h : X Ñ Y , define

Then, for any probability measure D over X ˆ Y , we can define

Definition H.3. S Ď X ˆ Y is said to be a an ε -net ( ε P p 0 , 1 q ) for H Ď Y X with respect to a distribution D over X ˆ Y if for any h P H ,

For any integer n P N , set Z , and T ' p z 1 , . . . , z n q P Z n , we say z P T if z ' z i for some i P r n s and use | T | to denote the length of the sequence T . For notational convenience, we use H to also denote an empty sequence (a sequence of length 0). For any subset E Ď Z , we use T X E ' E X T to denote the subsequence of T consisting of all elements in E , i.e., for I : ' t i P r n s : z i P E u ,

Then, we have | T X E | ' | E X T | ' | I | .

Proposition H.4. For any H Ď Y X with dim G p H q ' d , any H -realizable distribution D , any δ P p 0 , 1 s , any n P N , and any ERM algorithm A , consider S n ' D n . With probability at least 1 ´ δ , we have

Proof. For any n ě 2 and ε P r 2 { n, 1 s , define

Let p S, T q ' D 2 n with S, T P p X ˆ Y q n . Since p S, T q P B 1 implies that S P B , we have

P pp S, T q P B 1 q ' E r ✶ p S,T qP B 1 ✶ S P B s ' E r ✶ S P B P pp S, T q P B 1 | S qs . On S P B , there exists h P H such that er D p h q ě ε and ER p h qX S ' H . Then, | ER p h qX T | ą εn { 2 implies that p S, T q P B 1 . It follows that

✶ S P B P pp S, T q P B 1 | S q ě ✶ S P B P p| ER p h q X T | ą εn { 2 | S q . Since T is independent of S , h is determined by S , and D p ER p h qq ' er D p h q ě ε on S P B , we know that on S P B , | ER p h q X T | follows the Binomial distribution B p n, er D p h qq conditional on S , and

✶ Thus, by Lemma H.8, since nε ě 2 , we have which implies that

By the definition of B 1 , we have

For any k P r m s and 1 ď i 1 ă ¨ ¨ ¨ ă i k ď m , we use r p i 1 ,...,i k q to denote a permutation of r where p x j , y j q appears in the i j -th position for all j P r k s , specifically, r t i 1 ,...,i k u ' p x σ p i q , y σ p i q q i Pr n s where σ p i j q : ' j for all j P r k s and p σ p l qq l Pr n szt i 1 ,...,i k u : ' p k ` 1 , . . . , m q . Then, for any i Ď r 2 n s with | i | ' n , we have

Since p S, T q ' D 2 n , we also have

where (26) follows from Lemma H.7 and (27) follows from (25). Finally, we have proved that

Since D is H -realizable and A is an ERM algorithm, we must have A p S n , H q P H and

almost surely. Moreover, by the definition of B , if S n R B , then ER p A p S n , H qq X S n ' H implies that er D p A p S n , H qq ă ε . Thus, we have

Solving 2 τ H p 2 n q 2 ´ εn { 2 ' δ , we get

where the last inequality follows from Lemma H.2. Note that 1 ^ ' 2 n ` log 2 p τ H p 2 n qq ` log 2 ` 2 δ ˘˘‰ ě 2 n , which implies that the above choice of ε is legitimate. Applying (28), we can conclude that with probability at least 1 ´ δ ,

Proposition H.5. There exists a learning algorithm A G such that for any H Ď Y X with dim G p H q ' d , any H -realizable distribution D , any δ P p 0 , 1 s , and any n P N , given S n ' D n , it holds with probability at least 1 ´ δ that

Proof. The algorithm A G is the algorithm S n ÞÑ Majority p ERM H p A p S n ; Hqqq defined in Hanneke [2016], where ERM H denotes an ERM algorithm on the concept class H . Applying the error rate of ERM algorithms proved in Proposition H.4 in the proof of Hanneke [2016, Theorem 2], we establish (29).

The above proposition immediately implies the following corollary on the expected error rate of the learning algorithm A G .

Corollary H.6. There exists a learning algorithm A G such that for any H Ď Y X with dim G p H q ' d , any H -realizable distribution D , and any n P N it holds that

Proof. According to Proposition H.5, there exists some constant C ą 0 such that for any δ P p 0 , 1 s , it holds with probability at least 1 ´ δ that

which implies that for any t ě C p d ` 1 q n ,

Since er D p A G p S n , H qq is nonnegative, we have

Lemma H.7. For any n P N and m P N Xr n, 2 n s , we have

We prove by induction on m . When m ' 2 n , we have

Suppose that (31) holds for some m P N Xr n ` 1 , 2 n s . Then, we have

Thus, by induction, (31) holds for any m P N Xr n, 2 n s .

Lemma H.8. For X ' B p n, p q , if np ě 2 , then

Proof. If np ą 8 , since E r X s ' np , by the multiplicative Chernoff bound, we have

For 6 ď pn ď 8 , we have 6 { n ď p ď 8 { n and n ě 6 . Thus,

Proof. Note that

For 2 ď np ď 8 , we have

Consider

Fixing p P p 0 , 1 s , we have x ě 6 { p and for x ě 6 { p ,

Thus, fixing p P p 0 , 1 s , f 3 p¨ , p q is a decreasing function on r 6 { p, 8q . Therefore, we have

which implies that

for

Since log z ´ z ` 1 ď 0 for z P p 0 , 1 s and g 1 p z q ' log z ´ z ` 1 for g p z q : ' z log p z q` 1 ´ z ´ 1 2 p 1 ´ z q 2 defined on z P p 0 , 1 s , we have

Plugging in z ' 1 ´ 6 { t for t ą 6 , we have

for z P p 0 , 1 s . Thus, we have

Then, defining

we have

which implies that f p t q increases with t for t ą 6 . Since

we know that g 3 p 6 { p, p q decreases with p P p 0 , 1 s . Thus,

Following the above steps, for and

Specifically, we have

Then, we can conclude that

it is easy to prove that for any i ' 0 , 1 , 2 , and p P p 0 , 1 s ,

Next, we consider the regime that 4 ď np ă 6 . Now, we have

Following the procedures in the previous case, it is not hard to verify that for i ' 0 , 1 , 2 ,

It implies that

Finally, we consider the regime that 2 ď np ă 4 . Now, we have

Following the previous procedures, it is not hard to verify that for i ' 0 , 1 ,

It implies that

In conclusion, if np ě 2 , then

H.2 Bounding the graph dimension

For any H Ď Y X of DS dimension dim p H q ' d ă 8 , sequence S ' p x 1 , . . . , x n q P X n , menu µ : X Ñt Y Ď Y : | Y | ď p u of size p P N with n P N , and d 1 P r n s , define

for all i P ✷ r n s d 1 which denotes the collection of all subsets of r n s of size d 1 . We have the following lemma. Lemma H.9. dim G p H S,µ,d 1 q ď p 2 log 2 p e q ` 4 qp 5 d log 2 p p q ` 2 d 1 q for any d 1 P r n s .

Proof. For any i P r n s d 1 , by Bendavid et al. [1995], Daniely and Shalev-Shwartz [2014], we have

For any j Ď r n s that is G-shattered by H S,µ, i , define j 1 : ' j z i . We have that | j 1 | ě | j | ´ d 1 and j 1 is G-shattered by H 1 S,µ, i , which immediately implies that | j 1 | ď dim G p H 1 S,µ, i q ď 5 d log 2 p p q and | j | ď | j 1 | ` d 1 ď 5 d log 2 p p q ` d 1 . It follows that

For any m P N and j ' t j 1 , . . . , j m u Ď r n s that is G-shattered by H S,µ,d 1 , there exists f : r m s Ñ Y such that for any K Ď r m s , there exists g P H S,µ,d 1 satisfying g p j k q ' f p k q for all k P K and g p j k q ‰ f p k q for all k P r m sz K . It follows that | H S,µ,d 1 p j q| ' 2 m . If m ą d 1 , by Lemma H.2, we have which implies that

m

ď p which implies that

2 log

2

p

e

q `

4

qp

d

1

`

d

q ' p

2 log

2

p

e

q `

4

qp

5

d

log

2

p

p

q `

Lemma H.10. If x ą 0 satisfies x ď a log 2 p x { a q ` b for some a, b ą 0 , then, we have x ď 2 a ` 2 b .

Proof. Define f p x q : ' x ´ a log 2 p x { a q ´ b for x ą 0 . Then, we have f 1 p x q ' 1 ´ a x log p 2 q , which implies that f decreases with x for x P p 0 , a { log p 2 qq and increases with x for x ą a { log p 2 q . Since 2 a ` 2 b ą a { log p 2 q , it suffices to prove that f p 2 a ` 2 b q ě 0 . Indeed,

Lemma H.11. If x ą 0 satisfies x ď a log 2 p x { a q ` b log 2 p x { b q ` c for some a, b, c ą 0 , then, we have x ď 4 a ` 4 b ` 2 c .

Proof. Since

1

2

1

d

q

,

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In our abstract and introduction, we make the main claims that this paper reduces the gap between the lower and upper bounds of the multiclass PAC sample complexity and propose two possible routes towards completely resolving the optimal sample complexity, which are the paper's major contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: As is shown in Theorem 1.9, though we have reduced a log factor, there is still an O p ? d log p d q log p d { ε qq gap between our upper and lower bounds of the multiclass PAC sample complexity. We leave the development of sharper sample complexity upper bound in Open Question 1 and Open Question 2.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For each of our theoretical results, we provide the full set of assumptions in its statement as well as a complete and correct proof in the appendices.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper is fully theoretical and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

4.1 If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

4.2 If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

4.3 If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

4.4 We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper is fully theoretical and does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper is fully theoretical and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper is fully theoretical and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper is fully theoretical and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not include experiments and poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents improved sample complexity upper and lower bounds for multiclass classification, shaving previous upper bounds to within a factor of $\\log(1/\\epsilon)$ from the conjectured optimal dependence on $\\epsilon$, and adding a dependence of $\\log(1/\\delta)$ to previous best lower bounds. To do so, the authors study multiclass classification through the lens of the more genearal problem of list-learning -- this generality allows for a theorem stating that the conjectured optimal dependence of $1/\\epsilon$ in the sample complexity holds given a conjecture about the sample complexity of list-learning. They further discuss the potential of technique they term ``pivot shifting'' to remove the extra log factor of $1/\\epsilon$ in the multi class complexity, showing that whenever pivoting shifting does not increase a characteristic combinatorial complexity measure by more than a constant factor, pivot shifting eliminates the extra log factor.\n\nSTRENGTHS:\nThe paper introduces a novel boosting algorithm for list-learning which enables the log improvement in $1/\\epsilon$, allowing for the effective application of list-learning machinery to the multiclass problem. It improves on previous best lower bounds via the addition of the usual dependence on the confidence parameter $\\log(1/\\delta)$.\n\nWEAKNESSES:\nOutside of Section 1, I found the presentation of the paper to be very weak. Certain paragraphs present hard to follow proof outlines (e.g. line 236 \"the standard leave-one-out argument\" is not something I am familiar with and is not explained in the main body). Section 3, which introduces the concept of pivot shifting, provides little intuition into the notation-heavy definitions provided, and cannot be understood without reference to definitions which only appear in the appendix.\n\nIt is also a bit unclear what the upside of introducing the generality of list-learning is. The second half of Theorem 2.11 (line 281) does not feel like particularly compelling evidence for the conjectured $1/\\epsilon$ dependence, given that it relies on an analogous intuitive conjecture on the sample complexity of the more general problem of list learning.\n\nQUESTIONS:\n1) Is there a particularly compelling reason to believe in the existence of some $\\mathcal{A}_{goodlist}$ in Theorem 2.11? What do see as the advantage of thinking about multi-class learning through this extra layer of generality?  \n\n2) Could you elaborate on the $\\log(1/\\delta)$ factor in the lower bound (Theorem 2.5) -- does the proof technique follow that of Charikar and Pabbaraju [2023], or is this a different construction?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFor the problem of analyzing the optimal PAC sample complexity for multi-class learning, two possible routes to induce the improved sample complexity and error rate are proposed. On the one hand, benefiting from the reduction from multi-class learning to list learning and the boosting technique, the dependence of the corresponding upper bounds on the sample size is improved by a logarithmic factor. On the other hand, the introduction of \"pivot shifting\" shows its potential ability to improve the sample complexity of multi-class learning.\n\nSTRENGTHS:\n1. Based on the Daniely-Shalev-Shwartz (DS) dimension, using the reduction and the boosting technique, the explicit lower bound on the sample complexity of multi-class learning is derived, and the upper bound of the error rate for multi-class learning is improved by a logarithmic factor.\n2. In order to obtain the corresponding optimal theoretical bounds, the implementation limitations of some relevant assumptions are formalized as open problems and discussed in detail, i.e., the construction of the specific list learners and the impact of pivot shifting on DS dimension.\n\nWEAKNESSES:\nThe theoretical analysis in this paper improves the existing theoretical results for multi-class learning on the DS dimension, but the relationship between the obtained theoretical bounds and other data-independent complexity bounds based on combinatorial dimensions, e.g., the graph dimension, the Natarajan dimension and its scale-sensitive analog, is not discussed and compared, which is not conducive to demonstrating the improvement of these theoretical results from a macro perspective. In addition, existing theoretical results on multi-class learning show that data-independent generalization bounds often depend on the number of classes, and the best (weakest) known dependency is logarithmic. The results in this paper focus on improving the dependency on the number of samples, and do not explicitly discuss and analyze the dependency on the number of classes. These related issues require further analysis and explanation.\n\nQUESTIONS:\nPlease refer to the Weaknesses for details.\n\nLIMITATIONS:\nThis work does not seem to have any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work is focused on improving sample complexity upper and lower bounds on multiclass classifcation for a general hypothesis class. Their bounds are given in terms of the so-called DS-dimension of the hypothesis class, which can be viewed as a generalization of the VC dimension to the multiclass setting. Their bounds improve on the best known upper bounds and leave a factor of $O(\\sqrt{d})$ between their upper and lower bounds on the sample complexity.\n\nTheir main idea is to apply a reduction to list learning, where the learner is allowed to output a set of $k$ potential labels for each $x$ and is evaluated based on whether any of the outputted labels are correct. In particular, they provide a multiclass algorithm that works by calling a list learning algorithm as a subroutine. The subsequent guaranteed loss then grows logarithmically with $k$, the number of used labels. \n\nTheir full algorithm consequently requires a list learner, and they provide one that adapts a previously known boosting technique for binary classification to multiclass list learners. To apply boosting, they define the majority vote of a set of lists as the set of labels are included in at least half of the lists. Note that the resulting list has size $2k$ if each of the lists have size $k$. \n\nCombining this with their above reduction results in their upper bound.\n\nSTRENGTHS:\nThis paper is a highly technical paper that makes significant progress on a classic and important open problem in learning theory. In addition to improving the current best bounds, it provides an approach for further improvements. In particular, their reduction implies that improvements on list learning will result in improvements on multiclass classification.\n\nWEAKNESSES:\nMany of the proof ideas are differed to the appendix. I think it would improve the presentation to include small \"proof intution\" sections following the theorems. While this would cost more space, I think this can be accounted for by moving the highly technical algorithm blocks into the appendix and merely giving summaries of what the algorithms do in English. \n\nOf course, this paper is \"incremental\" in that it merely improves existing bounds to an preexisting problem, but I do not see this as cause at all for rejection. The improvements are clearly non-trivial, and the problem being studied is of fundamental importance.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of analyzing the sample complexity of Multiclass PAC Learning. The key contributions of this paper are two-fold: (1) Give an improved upper bound for the sample complexity by a poly-log factor, and (2) give the first (formal) lower bound for the Multiclass PAC Learning problem, which matches that of binary concept class. The key idea is to use a reduction from multiclass learning to list learning, which can then be further analyzed using recent works on boosting algorithms for list learners. \n\nBesides, the paper also explores a (potential) alternative approach to analyze the sample complexity for a concept class in Multiclass PAC Learning by alternatively analyzing its corresponding one-inclusion graph. Following this route, if we can give an upper bound for the density of any concept class (defined by the average degree of its corresponding one-inclusion graph) by multiple of its DS dimensions, we can have a matching lower bound for the problem of Multi-class PAC Learning. They argue that this approach is promising by giving proof for the case where the DS dimension of the concept class is 1.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. Motivation and key results in the main paper are presented clearly and concisely, though I would also want the proofs in the Appendix to be discussed and commented on in the same way as in the main paper. I did not go through the proof very carefully, but it looks good at first glance. I would try to go through that in the details in the rebuttal phase. \n\nI like the presentation of this paper: first presenting the improvement on the sample complexity of Multiclass PAC Learning using a technique, and then proposing an alternative view that can potentially lead to a matching upper bound. This gives good insights to the readers, not simply about the sample complexity improvements, but also about the problem itself. \n\nOverall, I think this is a good paper and vote to accept it.\n\nWEAKNESSES:\nThere is not much to comment on the weaknesses of this paper. A fastidious reviewer might argue that this paper is a collection of many good but not-so-strong results (in the sense that the list learner techniques used for the proofs of lower and upper-bound using are not completely new), but I think it is good enough for me. \n\nSome minor comments:\n1. It is good to include a proof sketch for every result presented in the Appendix, as well as the high-level intuitions behind the results. Though the statements of key results are clear, readers like us would like to see what is actually going on behind the proofs and would appreciate it if the authors could break down the steps for better readability.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

comments: SUMMARY:
This paper gives better bounds on the complexity of multiclass learning using DS-dimension and provides a lower bound. The improvement is relatively minor.

STRENGTHS:
The main strength of this paper is that it addresses one of the most fundamental problems in learning theory -- the complexity of multiclass learning and provides an improved bound on the sample complexity.

WEAKNESSES:
The writing of the paper and the presentation style can be improved substantially. The technical summary is way too dense and the comparison with the previous work is very narrow.

The technical improvement is rather weak (some log factors) and doesn't address the main gap (polynomial gap in d).

QUESTIONS:
-- I am not convinced that DS-dimension is the right quantity to look at here. Don't we already have optimal bounds for the sample complexity of multiclass learning in terms or Graph Dimension (see Daniely and Shalev-Shwartz)? Can you please explain how your results compare to the Graph Dimension results?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-23