Title: Derandomizing Multi-Distribution Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: pac learning

multi-distribution

derandomization

computational efficiency

discrepancy minimization
Abstract: Multi-distribution or collaborative learning involves learning a single predictor that works well across multiple data distributions, using samples from each during training. Recent research on multi-distribution learning, focusing on binary loss and finite VC dimension classes, has shown near-optimal sample complexity that is achieved with oracle efficient algorithms. That is, these algorithms are computationally efficient given an efficient ERM for the class. Unlike in classical PAC learning, where the optimal sample complexity is achieved with deterministic predictors, current multi-distribution learning algorithms output randomized predictors. This raises the question: can these algorithms be derandomized to produce a deterministic predictor for multiple distributions? Through a reduction to discrepancy minimization, we show that derandomizing multi-distribution learning is computationally hard, even when ERM is computationally efficient. On the positive side, we identify a structural condition enabling an efficient black-box reduction, converting existing randomized multi-distribution predictors into deterministic ones.

Derandomizing Multi-Distribution Learning

Abstract

Multi-distribution or collaborative learning involves learning a single predictor that works well across multiple data distributions, using samples from each during training. Recent research on multi-distribution learning, focusing on binary loss and finite VC dimension classes, has shown near-optimal sample complexity that is achieved with oracle efficient algorithms. That is, these algorithms are computationally efficient given an efficient ERM for the class. Unlike in classical PAC learning, where the optimal sample complexity is achieved with deterministic predictors, current multi-distribution learning algorithms output randomized predictors. This raises the question: can these algorithms be derandomized to produce a deterministic predictor for multiple distributions? Through a reduction to discrepancy minimization, we show that derandomizing multi-distribution learning is computationally hard, even when ERM is computationally efficient. On the positive side, we identify a structural condition enabling an efficient black-box reduction, converting existing randomized multi-distribution predictors into deterministic ones.

1 Introduction

We consider the problem of multi-distribution learning where there are k unknown data distributions P = {D 1 , . . . , D k } over X × {1 , 1 } , where X is an input domain and {-1 , 1 } are the possible labels. The goal is to learn a classifier f : X → {1 , 1 } that satisfies

̸

Here H ⊆ {1 , 1 } X is the benchmark hypothesis class of VC-dimension d that the learner competes against, and min h ∈H max i er D i ( h ) is the optimal worst-case error that can be achieved with classifiers from H . The framework of multi-distribution learning, introduced by Haghtalab et al. [10], is a natural generalization of agnostic PAC learning [22, 21, 5], and captures several important applications such as min-max fairness [12, 18, 15, 8, 20], and group distributionally robust optimization [16].

In the realizable setting, where min h ∈H er P ( h ) = 0 , there is a learning algorithm using ˜ O (( d + k ) /ε ) samples to produce such a deterministic classifier f , see e.g., the works [4, 7, 13]. Here, and throughout the paper, ˜ O hides terms that are poly ln( dk/ ( εδ )) .

In the more challenging agnostic setting, where OPT := min h ∈H er P ( h ) is greater than 0 , recent works show that the sample complexity is ˜ O (( d + k ) /ε 2 ) [10, 2, 14, 23]. We refer the reader to Table 1 in [23] for a detailed sample complexity comparison of prior algorithms. Importantly, the guarantee provided by all existing algorithms is slightly different from the objective (1) above. Concretely, all previous algorithms do not produce a deterministic classifier f : X → {1 , 1 } , but instead output a distribution F over H , such that

Due to the fact that classical PAC bounds, which involve learning from a single distribution, are achieved using deterministic predictors it is somewhat unsatisfactory to always output a randomized predictor in the multi-distribution case. Observe that because, as in (2), we want optimal performance simultaneously for all distributions, even using a randomized algorithm is somewhat problematic. Indeed, assume that in practice we want to sample a single ˆ f according to F and use it as our predictor. Now, if we seek a guarantee like (1) for ˆ f , then the best we can guarantee from (2) is to use Markov's inequality and a union bound over all k distributions to ensure that

with probability at least 1 / 2 , which is, of course, too conservative. Let us also remark that there are examples of distributions F for which this is basically tight. Consider e.g. an input domain X = x 1 , . . . , x k and k hypotheses h 1 , . . . , h k such that h i ( x i ) = -1 and h i ( x j ) = 1 for j = i . Let D i be the distribution that returns ( x i , 1) with probability 1 . Then for the uniform distribution F = k -1 ∑ i h i over classifiers, we have max i E f ∼ F [er D i ( f )] = 1 /k , but for any single f in the support of F , we have er P ( f ) = max i er D i ( f ) = 1 . The example also shows that for every fixed distribution D i , if we sample an f from F , then with probability 1 /k , its error exceeds the expectation by a factor k for that distribution D i . There may thus be a large gap between the guarantees of a deterministic and randomized classifier, i.e. the bounds in (1) and (2) are quite different.

̸

The main focus of our work, is on replacing the random classifiers in previous works on agnostic multi-distribution learning by deterministic classifiers and understanding the inherent complexity of doing so. In particular, we are interested in understanding any inherent statistical or computational gaps in multi-distribution learning between deterministic classifiers and randomized classifiers.

Our contributions

Our first contribution is a strong negative result towards derandomizing previous classifiers. Recall that the complexity class BPP denotes bounded-error probabilistic polynomial time 2 . That is, problems that have polynomial time randomized algorithms that are correct with probability at least 2 / 3 on every input. It is conjectured that P = BPP and thus most likely BPP = NP . Recall that a set of n points is shattered if each of the 2 n possible labelings of the points can be realized by some h ∈ H . Our negative result is then the following.

̸

̸

Theorem 1. If BPP = NP , then as n = min { d, k, 1 /ε } tends to infinity, for every hypothesis class H of VC-dimension d for which one can find n points shattered by H in polynomial time, any multidistribution learning algorithm for H that on the set of k input distributions P = {D 1 , . . . , D k } with probability at least 2 / 3 produces a deterministic classifier f : X → {1 , 1 } with er P ( f ) ≤ min h ∈H er P ( h ) + ε , must have either n ω (1) (i.e. super-polynomial) training time, or f has n ω (1) evaluation time.

We remark that this computational hardness result holds even when the class H admits efficient Empirical Risk Minimization (ERM), and even when the distributions are known to the learning algorithm. This highlights that the hardness stems not from the need to sample from the underlying distributions nor from the hardness of ERM, but from the computational problem of deciding which label to assign the points of the input domain.

Note that the assumption in Theorem 1 that one can find a set of n shattered points in polynomial time is not restrictive. Finding such points is trivial for many H , i.e., simply choose 0 , e 1 , . . . , e d -1 ∈ R d -1 = X for linear classifiers with VC-dimension d . More generally, the standard result on the

class of classifiers induced by positive halfspaces in R d shows that this class has VC dimension d , and for any set of points such that at most d of its points are contained on a single hyperplane, any subset of size d of this set is shattered. Similar properties are also known for the classes induced by balls in R p and positive sets in the plane defined by polynomials of degree at most p -1 . See [9] for a detailed exposition of these examples.

While this might have been the end of the story, our NP-hardness proof fortunately highlights a path to circumventing the lower bound. In particular, the proof carefully uses data distributions D 1 , . . . , D k for which D i ( y | x ) varies between the distributions. Here D i ( y | x ) denotes the conditional distribution of the label y of a sample ( x, y ) given x ∈ X . We thus consider the following restricted version of collaborative learning in which D i ( y | x ) = D j ( y | x ) for all x, i, j . That is, the k different distributions may vary arbitrarily over X , but the label y of any x ∈ X follows the same distribution for all D i . As a particular model of label consistent learning , one may think of a deterministic labeling setup where it is assumed that there is f ⋆ : X → { 1 , -1 } such that across all distributions y = f ⋆ ( x ) , while no assumption is made that f ⋆ belongs to H . Remarkably, in terms of sample complexity, in the case of a single distribution, the case of deterministic labeling is almost as hard as the general agnostic case as shown in [3]. Thus, we believe our label-consistent multi-distribution learning setup is quite natural and interesting.

Furthermore, this restriction turns out to be sufficient for derandomizing multi-distribution learning algorithms. In particular, we give a new algorithm, Algorithm 1, that uses a randomized (i.e., an algorithm that outputs a randomized predictor given the training data) multi-distribution learning algorithm (like (2)) as a black-box, and produces from it a deterministic classifier, as in (1).

Theorem 2. For any finite domain X , if the data distributions D 1 , . . . , D k are label-consistent, then given a multi-distribution learning algorithm A that uses m ( k, d, OPT , ε, δ ) samples and t ( k, d, OPT , ε, δ ) training time to produce, with probability 1 -δ , a distribution F over classifiers from H satisfying max i E f ∼ F [er D i ( f )] ≤ OPT+ ε, Algorithm 1 produces with probability 1 -δ a classifier f : X → {1 , 1 } with er P ( f ) ≤ OPT+ ε with the sample complexity

Using the additional ideas in Section 3.1, the training time of Algorithm 1 is

If the evaluation time of hypotheses in H is bounded by s , then the evaluation time of the classifier f is bounded by

Note that several of the previous randomized multi-distribution learning algorithms are indeed computationally efficient as long as ERM is efficient over H . This includes the algorithm in [23] that has a near-optimal sample complexity of m ( k, d, OPT , ε, δ ) = ˜ O (( d + k ) /ε 2 ) with t ( k, d, OPT , ε, δ ) = poly( k, d, ε -1 , ln(1 /δ )) t ERM and | F | = poly( k, d, ε -1 , ln(1 /δ )) , where t ERM denotes the time complexity of ERM over H . Plugging this into Theorem 2 gives a polynomial time deterministic multi-distribution learning algorithm.

We view the restriction to finite domains X in Theorem 2 as rather mild, as any realistic implementation of a learning algorithm requires an input representation that can be stored on a computer. Moreover, our running time dependency on |X| is only logarithmic. Even so, in Section 3.2 we give some initially promising directions for extending our algorithm to infinite X .

Discussion of implications. Prior work has shown that in agnostic multi-distribution learning, a sample complexity of Ω( dk/ϵ 2 ) , which is worse than the optimal ˜ O (( d + k ) /ϵ 2 ) sample complexity, is unavoidable with proper learning algorithms, which are algorithms restricted to outputting a classifier in the class H [23, Theorem 18]. In contrast, our negative result in Theorem 1 implies that there is no sample-efficient and oracle-efficient multi-distribution learning algorithm that aggregates multiple ERM predictors in polynomial time. For example, our result rules out the simple majority-vote aggregation approach (which is feasible in the realizable setting when OPT = 0 ). Note, however, that this does not rule out the existence of computationally inefficient aggregation approaches to construct deterministic predictors. That is, putting computational efficiency aside, it is still an open question whether there exists a sample-efficient and oracle-efficient multi-distribution learning algorithm that outputs a deterministic predictor, and we know from the lower bound of Zhang et al. [23, Theorem 18] that this predictor must be improper.

2 Hardness of derandomization

In this section, we prove that it is NP-hard to derandomize multi-distribution learning in the most general setup of input distributions D i over X × {1 , 1 } . In particular, the hardness proof carefully exploits that different data distributions may assign different labels to the same x ∈ X .

Our NP-hardness proof goes via a reduction from Discrepancy Minimization. In Discrepancy Minimization, we are given as input an n × n matrix with 0 -1 entries. The goal is to find a 'coloring' z ∈ {-1 , 1 } n such that every entry of Az is as small as possible in absolute value. Formally, we seek to minimize ∥ Az ∥ ∞ . The seminal work by Charikar et al. [6] showed NP-hardness of computing the best coloring. In full details, their results are as follows.

Theorem 3 ([6]) . There is a constant c > 0 such that it is NP-hard to distinguish whether an input matrix A ∈ { 0 , 1 } n × n has ∥ Az ∥ 2 ≥ cn for all z ∈ {-1 , 1 } n , or whether there exists z ∈ {-1 , 1 } n with Az = 0 .

Since ∥ Az ∥ ∞ ≥ ∥ Az ∥ 2 / √ n , this similarly implies that it is NP-hard to distinguish whether all z have ∥ Az ∥ ∞ ≥ c √ n , or there is a z with Az = 0 .

Let us now use Theorem 3 to prove our hardness result, Theorem 1. We remark that NP-hardness is formally defined in a uniform model of computation where a Turing Machine takes an encoded input on a tape and decides language membership. As we believe our reduction is clear without going into such formalities, we have deferred a discussion of how to formalize multi-distribution learning in a uniform model of computation to Appendix 4.

Proof. Let n = min { d, k/ 2 , c 2 / (4 ε 2 ) } and let H be an arbitrary hypothesis set of VC-dimension d for which we can find a set of n points that are shattered by H in n O (1) time. This is possible due to our assumption.

Let A denote an arbitrary deterministic multi-distribution algorithm. Given a matrix A ∈ { 0 , 1 } n × n such that either ∥ Az ∥ ∞ ≥ c √ n for all z ∈ {-1 , 1 } n , or there exists a z ∈ {-1 , 1 } n with Az = 0 , we will now use A to correctly distinguish these two cases with probability at least 2 / 3 , thus concluding that the running time of A is super-polynomial unless BPP = NP .

Start by computing an arbitrary set x 1 , . . . , x n of n points that are shattered by H . Now define 2 n distributions D + 1 , D -1 , . . . , D + n , D -n . Distribution D + i and D -i are both defined from the i -th row of A . If m i denotes the number of ones in the i -th row of A , we let D + i return the sample ( x j , 1) with probability 1 /m i for each j with a i,j = 1 . The distribution D -i similarly returns ( x j , -1) with probability 1 /m i for each j with a i,j = 1 . Observe that these distributions can be described using n bits each.

Now consider running the multi-distribution learning algorithm A on distributions D + 1 , D -1 , . . . , D + n , D -n to obtain a deterministic classifier f : X → {-1 , 1 } . Evaluate f on x 1 , . . . , x n and compute er P ( f ) . This can be done trivially in polynomial time using the definitions of the distributions. If er P ( f ) < 1 / 2 + ε , then output that there exists z ∈ {-1 , 1 } n such that Az = 0 . Otherwise, output that no such z exists. Clearly this runs in polynomial time. It thus remains to argue correctness.

Consider first the case where there exists z ∈ {-1 , 1 } n with Az = 0 . Since z has inner product 0 with every row of A , it follows that it assigns 1 to precisely half of the non-zero entries of the i -th row and -1 to the remaining half. The labeling z of x 1 , . . . , x n thus has er D + i ( z ) = er D -i ( z ) = 1 / 2 and er P ( z ) = 1 / 2 . Furthermore, since x 1 , . . . , x n are shattered by H , it follows that min h ∈H er P ( h ) ≤ 1 / 2 . By correctness of A , it must hold with probability at least 2 / 3 that we correctly output that there exists z with Az = 0 .

Consider next the case that every z ∈ {-1 , 1 } n has ∥ Az ∥ ∞ ≥ c √ n . It follows that there is a row a i such that the vector v = ( f ( x 1 ) , . . . , f ( x n )) has | v T a i | ≥ c √ n . Let σ = sign( v T a i ) . Then

Since we chose n = min { d, k/ 2 , c 2 / (4 ε 2 ) } , we have c/ (2 √ n ) ≥ ε and thus we return with probability 1 that all z ∈ {-1 , 1 } n have ∥ Az ∥ ∞ ≥ c √ n .

Let us end by observing that the distributions used in the above hardness result have OPT ≥ 1 / 2 . The proof can be modified to prove lower bounds for smaller OPT by adding a dummy point x 0 and letting all distributions return ( x 0 , 1) with probability 1 -2OPT ′ and the points in the above distributions with probability 2OPT ′ /m i . This reduces the value of OPT to around OPT ′ . However, we also need to reduce n to min { d, k/ 2 , ( c · OPT ′ /ε ) 2 } . This agrees well with the fact that for realizable multi-distribution learning, i.e., OPT = 0 , it is in fact possible to compute a deterministic classifier in polynomial time.

3 Deterministic multi-distribution learner

In this section, we give our algorithm for derandomizing multi-distribution learners for labelconsistent distributions, i.e., we assume D i ( y | x ) = D j ( y | x ) for all i, j, x .

̸

We start by presenting the high level ideas of our algorithm. Recall that we defined OPT = min h ∈H er P ( h ) , where P = {D 1 , . . . , D k } . First, consider running any of the previous randomized multi-distribution learners, producing a distribution F over hypotheses in H satisfying max i E f ∼ F [er D i ( f )] ≤ OPT + ε/ 2 . Consider randomly rounding this distribution to a deterministic classifier ˆ f as follows: For every x ∈ X independently (recall that we focus on finite domains), sample an f ∼ F and let ˆ f ( x ) = f ( x ) . For any distribution D i , we clearly have E ˆ f [er D i ( ˆ f )] = E f ∼ F [er D i ( f )] ≤ OPT + ε/ 2 . However, as also discussed in the introduction, it is not clear that we can union bound over all k distributions and argue that er D i ( ˆ f ) ≤ OPT + ε for all of them simultaneously. Notice however that the independent choice of ˆ f ( x ) for each x gets us most of the way. Indeed, if we let Z x be a random variable (determined by ˆ f ( x ) ) giving Pr y ∼D ( y | x ) [ ˆ f ( x ) = y ] , then er D i ( ˆ f ) = ∑ x ∈X D i ( x ) Z x , where D i ( x ) denotes the probability of x under D i and D i ( y | x ) gives the conditional distribution of the label y given x . Now notice that D i ( x ) Z x is a random variable taking values in { (1 / 2 -| β x | ) D i ( x ) , (1 / 2 + | β x | ) D i ( x ) } where β x = Pr y ∼D 1 ( y | x ) [ y = 1] -1 / 2 denotes the bias of the label of x . Furthermore, these random variables are independent. We also have E [er D i ( ˆ f )] = E f ∼ F [er D i ( f )] . Thus by Hoeffding's inequality

Examining this expression closely, we observe that this probability is small if β x D i ( x ) is small for all x ∈ X .

Using this observation, our algorithm then starts by drawing ˜ O ( ε -2 ) samples from each distribution D i and collecting all x for which the fraction of 1 's and -1 's is so biased towards either 1 or -1 , that the majority label almost certainly equals sign( β x ) . We then let ˆ f ( x ) equal this majority label for all such x , and put these x into a set T .

What remains is all x / ∈ T . Here we show that these x have so little bias, i.e., β x D i ( x ) is so small, that the random rounding strategy above suffices. The full algorithm is shown as Algorithm 1.

Before giving the formal analysis of the algorithm, note that storing the classifier ˆ f is quite expensive, as we need to remember the random choice of ˆ f ( x ) for every x ∈ X \ T . This is one place where we

Algorithm 1: DETERMINISTICLEARNER( P , ε, δ, A )

Input: Distributions P = {D 1 , . . . , D k } . Precision ε > 0 , failure probability δ > 0 , randomized multi-distribution learner A .

Result: Classifier ˆ f : X → {1 , 1 } .

1 Let C > 0 be a large enough constant.

2 Let γ = Ck/ ( εδ ) .

3 Let T = ∅ .

4 Run A with P = {D 1 , . . . , D k } and H as input, precision ε/ 2 and failure probability δ/ 2 to obtain a distribution F over classifiers in H .

5 for i = 1 , . . . , k do

6 Draw m = C ln 2 ( γ ) /ε 2 samples { ( x j , y j ) } j m =1 from D i .

7

For every

ρ

i,x

= (

x

|{

∈ X \

j

:

x

j

T

=

such that

x

∧

y

j

n

i,x

:=

}| - |{

= 1

|{

j

:

j

:

x

j

x

j

=

=

x

x

∧

}|

y

j

>

=

0

, let

-

1

}|

)

/n i,x

.

8 For every x ∈ X \ T such that n i,x > 0 , if | ρ i,x | > √ ln( γ ) /n i,x , add x to T and let ˆ f ( x ) = sign( ρ i,x ) .

9 For every x ∈ X \ T , independently draw an f ∼ F and let ˆ f ( x ) = f ( x ) .

10 return ˆ f use the assumption that X is finite. Note however that even for finite X , storing |X| random choices to represent the classifier might be infeasible. Furthermore, the sampling of ˆ f ( x ) for every x also adds |X| to the running time, which is again too expensive. We propose a method for reducing the storage and running time requirement later in this section. For now, we analyse Algorithm 1 without worrying about |X| .

Analysis. In our analysis, we separately handle x ∈ T and x / ∈ T . The two technical results we need are stated next. First, define the bias β x of an x ∈ X as Pr y ∼D 1 ( y | x ) [ y = 1] -1 / 2 . We say that an x is heavily biased if

for at least one i , and lightly biased otherwise. Intuitively, our algorithm ensures that T contains all heavily biased x and that all predictions made on x ∈ T are correct. This is stated in the following

Lemma 4. It holds with probability at least 1 -δ/ 4 that every heavily biased x is in T , and furthermore, for every x ∈ T , we have ˆ f ( x ) = sign( β x ) .

Next, we also show that random rounding outside T suffices.

Lemma 5. Assume every heavily biased x is in T after the for-loop. Then with probability at least 1 -δ/ 4 over the random choice of ˆ f ( x ) with x ∈ X \ T , it holds for all i that

̸

̸

Before giving the proof of Lemma 4 and Lemma 5, let us use these two results to complete the proof of Theorem 2.

Proof of Theorem 2. From a union bound and Lemma 4 and Lemma 5, we have with probability 1 -δ , that all of the following hold

The invocation of A in step 1 of Algorithm 1 returns a distribution F with max i E f ∼ F [er D i ( f )] ≤ OPT+ ε/ 2 .

For every x ∈ T , we have ˆ f ( x ) = sign( β x ) .

For every distribution D i ,

̸

̸

Assume now that all of the above hold. We rewrite er P ( ˆ f ) by splitting the contributions to the error into x ∈ T and x / ∈ T ,

̸

̸

̸

Using that ˆ f ( x ) = sign( β x ) for x ∈ T , we have E ( x,y ) ∼D i [1 { x ∈ T ∧ ˆ f ( x ) = y } ] = min z ∈{-1 , 1 } T [ E D i [1 { x ∈ T ∧ z ( x ) = y } ] . Thus the above is bounded by

̸

̸

Since every f in the support of F is a deterministic classifier, we have

̸

We therefore have

This completes the proof of Theorem 2.

Proof of Lemma 4. We first define the two types of failures that may occur:

For every i and every x with β 2 x D i ( x ) > ε 2 / (8 ln(4 k/δ )) , let E 1 i,x denote the event that n i,x < ( C/ 2) β -2 x ln γ .

For every i , let E 2 i denote the event that there is an x with n i,x > 0 and | β x -ρ i,x / 2 | > √ ln( γ ) / (16 n i,x ) .

Assume first that none of the events occur. Consider a heavily biased x . Then there is an i for which β 2 x D i ( x ) > ε 2 / (8 ln(4 k/δ )) . Since E 1 i,x does not occur, we have n i,x ≥ ( C/ 2) β -2 x ln γ . Since E 2 i does not occur, we also have | β x -ρ i,x / 2 | ≤ √ ln( γ ) / (16 n i,x ) . Hence | ρ i,x | ≥ 2 | β x | -2 √ ln( γ ) / (16 n i,x ) . But | β x | ≥ √ ( C/ 2) ln( γ ) /n i,x and thus | ρ i,x | ≥ ( √ 2 C -1 / 2) √ ln( γ ) /n i,x . For C large enough, this is at least √ ln( γ ) /n i,x , which puts x in T during step 8 of Algorithm 1. Thus every heavily biased x is in T . Secondly, note that when an x is added to T in iteration i of the for-loop, we have | ρ i,x | > √ ln( γ ) /n i,x . Since E 2 i does not occur, we have | β x -ρ i,x / 2 | ≤ √ ln( γ ) / (16 n i,x ) . But this implies β x ∈ [ ρ i,x / 2 -√ ln( γ ) / (16 n i,x ) , ρ i,x / 2 + √ ln( γ ) / (16 n i,x )] . Since | ρ i,x | > √ ln( γ ) /n i,x , every number in this interval has the same sign as ρ i,x , i.e. ˆ f ( x ) = sign( ρ i,x ) = sign( β x ) . Thus what remains is to bound the probability of these events.

For E 1 i,x , fix an i and x with β 2 x D i ( x ) > ε 2 / (8 ln(4 k/δ )) , we have

For C large enough, we get from a Chernoff bound that Pr[ E 1 i,x ] = Pr[ n i,x < ( C/ 2) β -2 x ln γ ] < γ -2 .

For E 2 i , let us first condition on an outcome of the values n i,x for all x . Then for every x , we have that p i,x := |{ j : x j = x ∧ y j = 1 }| - |{ j : x j = x ∧ y j = -1 }| is distributed as the sum of n i,x independent -1 / 1 random variables taking the value 1 with probability β x +1 / 2 . Hence E [ p i,x ] = 2 β x n i,x . Since ρ i,x = p i,x /n i,x , it follows from Hoeffding's inequality that

̸

̸

̸

̸

For any fixed values n i,x , there are at most m distinct x with a non-zero n i,x . A union bound over all of them implies Pr[ E 2 i | n i,x ] ≤ 2 mγ -2 . Since this upper bound holds for any outcome of the n i,x , we have also Pr[ E 2 i ] ≤ 2 mγ -2 .

We now observe that for every i , there are at most ε -2 8 ln(4 k/δ ) distinct x with β 2 x D i ( x ) > ε 2 / (8 ln(4 k/δ )) . Hence Pr[ ∪ x E 1 i,x ] ≤ ε -2 8 ln(4 k/δ ) γ -2 . A union bound over all i finally implies

Since γ = Ck/ ( εδ ) and m = C ln 2 ( γ ) /ε 2 , we have for large enough C that this probability is bounded by δ/ 4 .

̸

Proof of Lemma 5. Fix a distribution D i . Observe that for any x ∈ X \ T , we have that the distribution of ˆ f ( x ) is the same as f ( x ) for f ∼ F . Hence E ˆ f [ E ( x,y ) ∼D i [1 { x / ∈ T ∧ ˆ f ( x ) = y } ]] = E f ∼ F [ E ( x,y ) ∼D i [1 { x / ∈ T ∧ f ( x ) = y } ]] . Denote this expectation by µ . If we let Z x be the random variable (as a function of ˆ f ( x ) ) taking the value Pr y ∼D i ( y | x ) [ ˆ f ( x ) = y ] , then

̸

̸

Observe that Z x is either 1 / 2 -| β x | or 1 / 2 + | β x | , depending on whether ˆ f ( x ) = sign( β x ) or not. Hence D i ( x ) Z x ∈ [ D i ( x )(1 / 2 -| β x | ) , D i ( x )(1 / 2 + | β x | )] and the Z x are independent. We thus get from Hoeffding's inequality and that x / ∈ T are lightly biased that

A union bound over all D i completes the proof.

3.1 Reducing storage and time

The above description of Algorithm 1 requires the storage of an independent random choice of ˆ f ( x ) for every x ∈ X . This is infeasible for large X , both in terms of space usage and the time needed for making these random choices. Instead, we can reduce the storage requirements by using an r -wise independent hash function q : X → Y for a sufficiently large output domain Y to make the random rounding. Recall that an r -wise independent hash function hashes any set of up to r distinct keys x 1 , . . . , x r independently and uniformly at random into Y . Such a hash function can be implemented in space O ( r ln( |X||Y| )) bits and evaluated in time ˜ O ( r ln( |X||Y| )) by e.g., interpreting an x ∈ X as an index into [ |X| ] = { 0 , . . . , |X| 1 } and letting q ( x ) = ∑ r -1 i =0 α i x i (mod p ) for a prime p = |Y| > |X| and the α i independent and uniformly random in [ p ] . Using fast multiplication algorithms, q ( x ) can be evaluated in time ˜ O ( r ln( |X||Y| )) , even when ln( |X||Y| ) bits does not fit in a machine word. The time to sample the hash function is only O ( r ln |X||Y| ) (we just need the random coefficients of the polynomial).

Instead of storing ˆ f ( x ) for every x ∈ X \ T explicitly, the learning algorithm instead stores q and the distribution F . Given this information, we evaluate ˆ f ( x ) by computing q ( x ) and letting ˆ f ( x ) = 1 if q ( x ) ≤ Pr f ∼ F [ f ( x ) = 1] |Y| 1 and -1 otherwise. Since q ( x ) is uniform over Y for any x , we have Pr[ ˆ f ( x ) = 1] = ⌊ Pr f ∼ F [ f ( x ) = 1] |Y|⌋ / |Y| . This probability satisfies Pr f ∼ F [ f ( x ) = 1] -1 / |Y| ≤ Pr f ∼ F [ f ( x ) = 1] ≤ Pr f ∼ F [ f ( x ) = 1] and is thus almost the same rounding probability as in Algorithm 1. Since previous multi-distribution learning algorithms also store F , this only adds O ( r ln( |X||Y| )) bits to the storage.

What remains is to determine an r and |Y| for which this is sufficient for the guarantees of Algorithm 1. We will show that r = 2ln(4 k/δ ) and |Y| = Θ( ε -3 ln( k/δ )) suffices if we increase the sample

̸

complexity of Algorithm 1 by a logarithmic factor. Observe that the O ( r ln( |X| ln( k/δ ) /ε )) extra bits is only proportional to storing O (ln( k/δ )) samples from X , provided that ln( k/δ ) /ε is no larger than a polynomial in |X| . The space overhead is thus very minor.

We only give an outline of how to modify the proof in the previous section to work with r -wise independence as it follows the previous proof rather uneventfully. First, redefine the threshold for being heavily biased to β 2 x D i ( x ) > ε 2 / ( C ′ ln 2 (4 k/δ )) for large enough constant C ′ .

For the proof of Lemma 4 to still go through, this requires us to increase m by a C ′ ln γ factor, i.e. to CC ′ ln 3 ( γ ) /ε 2 , and also increase γ by C ′ to CC ′ k/ ( εδ ) . Then the only change to the proof, is that we have an event E 1 i,x for every i and every x with β 2 x D i ( x ) > ε 2 / ( C ′ ln 2 (4 k/δ )) . Otherwise, all conditions in the events E 1 i,x and E 2 i remain the same. Thus the proof still goes through if we can argue Pr[ E 1 i,x ] ≤ γ -2 . So fix an i and x with β 2 x D i ( x ) > ε 2 / ( C ′ ln 2 (4 k/δ )) . Then E [ n i,x ] = D i ( x ) m>ε 2 m/ ( C ′ β 2 x ln 2 (4 k/δ )) > Cβ -2 x ln γ . This is the same lower bound on E [ n i,x ] as the previous proof and thus we can complete the steps. Finally, note that we finished the proof of Lemma 4 by a union bound. Here we needed kγ -2 ( ε -2 8 ln(4 k/δ ) + 2 m ) < δ/ 4 . This is still the case for our new m and γ .

Now for the proof of Lemma 5, we used Hoeffding's inequality. This requires the random rounding to be independent for different x . With our modified approach, the roundings are only r -wise independent and thus we need the following variant of Hoeffding's inequality for r -wise independent random variables

Theorem 6 ([17]) . Let Z 1 , . . . , Z n be a sequence of r -wise independent random variables for r ≥ 2 with | Z i -E [ Z i ] | ≤ 1 for all outcomes. Let Z = ∑ i Z i with E [ Z ] = µ and let σ 2 ( Z ) = ∑ i σ 2 ( Z i ) denote the variance of Z . Then the following holds for even r and any Q ≥ max { r, σ 2 ( Z ) } :

̸

If we repeat the proof of Lemma 5, define Z x as the random variable (as a function of the random choice of q ) taking the value Pr y ∼D i ( y | x ) [ ˆ f ( x ) = y ] . Note that Z x ∈ { 1 / 2 -| β x | , 1 / 2 + | β x |} . This also implies that | Z x -E [ Z x ] | ≤ 2 | β x | for all outcomes of Z x . When all heavily biased x are in T , we have β 2 x D i ( x ) ≤ ε 2 / ( C ′ ln 2 (4 k/δ )) for all x / ∈ T . This implies | β x | ≤ ε/ (ln(4 k/δ ) √ C ′ D i ( x )) . Now let α = 2 ε/ (ln(4 k/δ ) √ C ′ ) . Then

̸

The random variable D i ( x ) Z x /α thus satisfies |D i ( x ) Z x /α -E [ D i ( x ) Z x /α ] | ≤ 2 D i ( x ) | β x | /α ≤ √ D i ( x ) ≤ 1 for all outcomes. This also gives us σ 2 ( D i ( x ) Z x /α ) ≤ D i ( x ) and thus

Now consider the expected value (with a ± b = [ a -b, a + b ] )

̸

̸

̸

̸

Letting µ = E f ∼ F [ E ( x,y ) ∼D i [1 { x / ∈ T ∧ f ( x ) = y } ]] , we then have by Theorem 6 with Q = r that

Inserting T = ε/ (2 α ) and using r = 2ln(4 k/δ ) , |Y| ≥ 4 α 2 /ε gives ( T -α/ |Y| ) ≥ ε/ (4 α ) and thus finally implies

̸

̸

Here, the last inequality follows for C ′ large enough. Thus, if we increase the sample complexity to m ( k, d, OPT , ε/ 2 , δ/ 2) + O ( k ln 3 ( k/ ( εδ )) /ε 2 ) , then we may sample and store a hash function using only O (ln( n/δ ) ln( |X| ln( k/δ ) /ε )) extra bits and O (ln( n/δ ) ln( |X| ln( k/δ ) /ε )) time.

3.2 Infinite Input Domains

In the above presentation of our algorithm, we have assumed a finite input domain X . While we believe this is a very reasonable assumption, we here present some initial ideas for how this restrictions might be circumvented.

Assume that the black-box randomized multi-distribution learner A always outputs a distribution F over a finite number of classifiers in H . Let m be an upper bound on the size of the support. Then since H has VC-dimension d , the dual VC-dimension is at most 2 d [1]. By Sauer-Shelah, this implies that the number of distinct ways x ∈ X may be labeled by the support of F is bounded ( m 2 d +1 ) , i.e. finite. We believe that treating just the distinct ways x is labeled by the hypotheses in the support should be sufficient to recover our results for finite X .

Acknowledgments

Kasper Green Larsen is co-funded by a DFF Sapere Aude Research Leader Grant No. 9064-00068B by the Independent Research Fund Denmark and co-funded by the European Union (ERC, TUCLA, 101125203). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. Omar Montasser was supported by a FODSI-Simons postdoctoral fellowship at UC Berkeley.

References

P. Assouad. Densité et dimension. Annales de l'Institut Fourier , 33:233-282, 1983.

P. Awasthi, N. Haghtalab, and E. Zhao. Open problem: The sample complexity of multidistribution learning for VC classes. In G. Neu and L. Rosasco, editors, The Thirty Sixth Annual Conference on Learning Theory, COLT 2023, 12-15 July 2023, Bangalore, India , volume 195 of Proceedings of Machine Learning Research , pages 5943-5949. PMLR, 2023.

S. Ben-David and R. Urner. The sample complexity of agnostic learning under deterministic labels. In Conference on Learning Theory , pages 527-542. PMLR, 2014.

A. Blum, N. Haghtalab, A. D. Procaccia, and M. Qiao. Collaborative PAC learning. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 2392-2401, 2017.

A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth. Learnability and the vapnikchervonenkis dimension. J. ACM , 36(4):929-965, 1989.

M. Charikar, A. Newman, and A. Nikolov. Tight hardness results for minimizing discrepancy. In D. Randall, editor, Proceedings of the Twenty-Second Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2011, San Francisco, California, USA, January 23-25, 2011 , pages 1607-1614. SIAM, 2011.

J. Chen, Q. Zhang, and Y. Zhou. Tight bounds for collaborative PAC learning via multiplicative weights. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 3602-3611, 2018.

E. Diana, W. Gill, M. Kearns, K. Kenthapadi, and A. Roth. Minimax group fairness: Algorithms and experiments. In M. Fourcade, B. Kuipers, S. Lazar, and D. K. Mulligan, editors, AIES '21: AAAI/ACM Conference on AI, Ethics, and Society, Virtual Event, USA, May 19-21, 2021 , pages 66-76. ACM, 2021.

S. Floyd and M. K. Warmuth. Sample compression, learnability, and the vapnik-chervonenkis dimension. Mach. Learn. , 21(3):269-304, 1995.

N. Haghtalab, M. I. Jordan, and E. Zhao. On-demand sampling: Learning optimally from multiple distributions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022 , 2022.

M. J. Kearns and U. V. Vazirani. An Introduction to Computational Learning Theory . MIT Press, Cambridge, MA, USA, 1994.

M. Mohri, G. Sivek, and A. T. Suresh. Agnostic federated learning. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 4615-4625. PMLR, 2019.

H. L. Nguyen and L. Zakynthinou. Improved algorithms for collaborative PAC learning. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 7642-7650, 2018.

B. Peng. The sample complexity of multi-distribution learning. CoRR , abs/2312.04027, 2023.

G. N. Rothblum and G. Yona. Multi-group agnostic PAC learnability. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 9107-9115. PMLR, 2021.

S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

J. P. Schmidt, A. Siegel, and A. Srinivasan. Chernoff-hoeffding bounds for applications with limited independence. SIAM Journal on Discrete Mathematics , 8(2):223-250, 1995.

S. Shekhar, G. Fields, M. Ghavamzadeh, and T. Javidi. Adaptive sampling for minimax fair classification. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 24535-24544, 2021.

M. Sipser. Introduction to the theory of computation. SIGACT News , 27(1):27-29, 1996.

C. J. Tosh and D. Hsu. Simple and near-optimal algorithms for hidden stratification and multi-group learning. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 21633-21657. PMLR, 2022.

L. G. Valiant. A theory of the learnable. Commun. ACM , 27(11):1134-1142, 1984.

V. Vapnik and A. Chervonenkis. Theory of Pattern Recognition . Nauka, Moscow, 1974.

Z. Zhang, W. Zhan, Y. Chen, S. S. Du, and J. D. Lee. Optimal multi-distribution learning. CoRR , abs/2312.05134, 2023.

4 Uniform Model of Computation

For a fully formalized NP-hardness proof, we technically need to define an input encoding of a multi-distribution learning problem and argue that the sampling steps may be simulated by a Turing Machine. Furthermore, details such as whether the hypothesis set H is part of the input or known to the algorithm also needs to be formalized. In this section, we discuss various choices one could make. We note that similar discussions and formalizations of learning in a uniform model of computation has been carefully carried out in classic learning theory books [11].

First, we find it most natural that H is part of the learning problem, i.e. not an input to the algorithm, but is allowed to be "hard-coded" into the algorithm. This is the best match to standard learning problems, where e.g. the Support Vector Machine learning algorithm, or Logistic Regression via gradient descent, knows that we are working with linear models. Similarly, the input domain seems best modeled by letting it be known to the algorithm. One tweak could be that if the input is d -dimensional vectors, then d could be part of the input to the algorithm. This again matches how most natural learning algorithms work for arbitrary d (and our proof needs d to grow for our n to grow).

Nowregarding modeling multi-distribution learning, we find that the following uniform computational model most accurately matches what the community thinks of as multi-distribution learning (here stated for the input domain being n -dimensional vectors and the hypothesis set being linear models).

A solution to multi-distribution learning with linear models, is a special Turing machine M . M receives as input a number n on the input tape. In addition to a standard input/output tape and a tape with random bits, M has a "sample"-tape, a "target distribution"-tape and a special "sample"-state. When M enters the "sample"-state, the bits on the "target distribution" tape is interpreted as an index in i and the contents of the "sample"-tape is replaced by a binary description of a fresh sample from a distribution D i ( D i is only accessible through the "sample"-state). A natural assumption here would be that D i is only supported over points with integer coordinates bounded by n in magnitude. This gives a natural binary representation of each sample using n log n bits, plus one bit for the label.

M runs until terminating in a special halt state, with the promise that regardless of what n distributions D 1 , . . . , D n over the input domain that are used for generating samples in the "sample"-state, it holds with probability at least 2 / 3 over the samples and the random bits on the tape, that the output tape contains a binary encoding of a hyperplane with error at most τ +1 /n for every distribution D i . Abit more generally, we could also let it terminate with an encoding of a Turing machine on its output tape. That Turing machine, upon receiving the encoding of n and an n -dimensional point on its input tape, outputs a prediction on its tape. This allows more general hypotheses than just outputting something from H .

The above special states and tapes are introduced to most accurately represent multi-distribution learning. Now observe that our reduction from discrepancy minimization still goes through. Given

such a special Turing machine M for multi-distribution learning, observe that we can obtain a standard (randomized) Turing machine M ′ for discrepancy minimization from it. Concretely, in discrepancy minimization, the input is the integer n and an n × n binary matrix A . As mentioned in our reduction, we can easily compute n shattered points for linear models, e.g. just the standard basis e 1 , . . . , e n . Now do as in our reduction and interpret each row of A as two distributions over e 1 , . . . , e n . M ′ can now simulate the "sample"-state, "sample"-tape and "target distribution" tape of M , as it can itself use its random tape to generate samples from the distributions. In this way, M ′ can simulate M without the need for special tapes and states, and by the guarantees of M (as in our reduction), it can distinguish whether A has discrepancy 0 or Ω( √ n ) by using the final output hypothesis of M and evaluating it on e 1 , . . . , e n and computing the error on each of the (known) distributions D i obtained from the input matrix A .

Note that the reduction would also hold if we rephrased multi-distribution learning such that the algorithm receives some binary encoding of D 1 , . . . , D n as input. This would make the reduction even more straight-forward, as we need not worry about samples. However, we feel the above definition with a special state and tapes for sampling more accurately represent multi-distribution learning from a learning theoretic perspective. We thus prefer a slightly more complicated reduction as above to better model the problem.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The theoretical claims are stated in the introduction and abstract. They match the contribution of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper is theoretical, so given the assumptions of the claims, there are no limitations. However, we discuss that our paper does not exclude the existence of inefficient deterministic predictors (see introduction).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper includes all proofs of all claims in the main text.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper is theoretical, and we have no experiments, data or code in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper is theoretical, and we have no experiments, data or code in the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper is theoretical, and we have no experiments, data or code in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper is theoretical, and we have no experiments, data or code in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper is theoretical, and we have no experiments, data or code in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We see no violations of the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The result of the paper is theoretical/foundational research, so we have no experiments, data or code in the paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper is theoretical, and we have no experiments, data or code in the paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper is theoretical, and we have no experiments, data or code in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper is theoretical, and we have no experiments, data or code in the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper is theoretical, and we have no experiments, data or code in the paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper is theoretical, and we have no experiments, data or code in the paper. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMulti-distribution learning is an extension of the agnostic learning setup which defined as follows. Given a collection of k distributions $(D_1, …, D_k)$ over the domain $\\mathcal{X} \\times \\\\{\\pm 1\\\\}$ and baseline hypothesis class $\\mathcal{H}$ find a (potentially randomized) classifier $f: \\mathcal{X} \\to \\\\{\\pm 1\\\\}$ such that its *worst-case* loss $\\max_{i \\in [k]} \\mathbb{E}_{f \\sim \\mathcal{F}}L_i(f)$ is competitive with that of the best fixed classifier in $\\mathcal{H}$. Here, $L_i$ denotes the zero-one loss $L_i(f) = \\Pr_i[f(X) \\neq Y]$ where the probability $\\Pr_i$ defined is over $(X, Y) \\sim D_i$.\n\nPrevious work has shown that one can construct *randomized* classifiers (i.e., a distribution $\\mathcal{F}$ over classifiers) that succeeds in multi-distribution learning with sample complexity proportional to the VC dimension of the baseline class $\\mathcal{H}$. This work addresses the question of whether it is possible to efficiently \"derandomize\" such classifiers to achieve multi-distribution learning with a *deterministic* classifier. The main contributions are twofold.\n\n1. Learning a deterministic classifier in the multi-distribution learning setting is *computationally* hard since **discrepancy minimization**, which is an NP-hard problem, reduces to multi-distribution learning. \n2. If we assume that the given distributions $(D_1,...,D_k)$ share a common conditional distribution $f^*(x) = \\Pr[Y = 1|X=x]$, then derandomization is possible by \"essentially\" (see comment below) labeling *each* $x \\in \\mathcal{X}$ with independently drawn classifiers $f \\sim \\mathcal{F}$.\n\n\"essentially\": the statement is true for marginal distributions with high min-entropy. For distributions with \"heavy\" atoms, there is a simple fix; if $x$ is a heavy element, estimate the conditional probability $f^*(x) = \\Pr[Y=1|X=x]$ with sufficient confidence (which is possible since $x$ is a heavy element) and label it with $\\mathrm{sign}(f^*(x))$.\n\nSTRENGTHS:\nThe main results of this work stem from simple but powerful observations. The insistence on learning *deterministic* classifiers turns the multi-distribution learning problem into a combinatorial one, which already suggests that some NP-hardness result could be right around the corner. The authors drive this intuition home by connecting the multi-distribution learning to discrepancy minimization, which is known to be NP-hard. This connection, though simple to establish, appears to be fundamental and may lead to exciting opportunities for cross-pollination between learning theory and discrepancy theory. Below is a technical description highlighting the key insight of the main reduction.\n\n**Reduction from discrepancy minimization.** Discrepancy minimization is defined as follows. Let $c > 0$ be any fixed constant. Given an $n \\times n$ matrix $A \\in \\\\{0,1\\\\}^{n \\times n}$ distinguish between the case where 1) there exists $z \\in \\\\{\\pm 1\\\\}^n$ such that $Az = \\mathbf{0}$ and 2) $||Az||_\\infty \\ge c \\sqrt{n}$ for all $z \\in \\\\{\\pm 1\\\\}^n$. For some constant $c > 0$, this problem is NP-hard.\n\nThe authors' key insight is the following. Consider two \"antithetical\" distributions $D_1, D_{-1}$ where $D_1, D_{-1}$ share the same marginal distribution $\\mu_1$ supported on an $n$-point set $\\\\{x_1,...,x_n\\\\}$ but $D_1$ assigns all the mass on the set $\\\\{(x,1) | x \\in \\\\{x_1,...,x_n\\\\}\\\\}$ and $D_{-1}$ only on $\\\\{(x,-1) | x \\in \\\\{x_1,...,x_n\\\\}\\\\}$. Then, the worst-case zero-one loss achievable w.r.t. $D_1,D_{-1}$ by *any* classifier is 1/2. Moreover, any *deterministic* classifier that achieves the optimal value OPT = 1/2 \"perfectly balances\" the $n$ points $x_1,...,x_n$ w.r.t. the marginal measure $\\mu_1$. Let $A_i \\in \\\\{0,1\\\\}^n$ be the $i$-th row of the discrepancy instance $A \\in \\\\{0,1\\\\}^{n \\times n}$. If there exists $z \\in \\\\{\\pm 1\\\\}^n$ such that $A z = \\mathbf{0}$, then $A_i z = 0$. We can convert each row $A_i$ into a pair of distributions $D_{i}, D_{-i}$ such that OPT=1/2 if and only if there exists $z \\in \\\\{\\pm 1\\\\}^n$ s.t. $Az = \\mathbf{0}$.\n\nNote the minimal role of the baseline class $\\mathcal{H}$. All we require is that $\\mathcal{H}$ shatter the $n$-point support $\\{x_1,...,x_n\\}$. A sufficient condition that satisfies this is $\\mathrm{VCdim}(\\mathcal{H}) > n$.\n\nWEAKNESSES:\nWhile this work constitutes a solid contribution to learning theory, it falls a little short in the following aspects.\n\n**Formal definition of a mutli-distribution learning in the uniform computational model?** While the key idea behind the reduction from discrepancy minimization is clear, some formalities in making the reduction rigorous seems to be missing. NP-hardness is a notion defined for uniform computational models (e.g., Turing machines). To use NP-hardness of discrepancy minimization as the basis for the hardness of multi-distribution learning, one needs to define what it means to solve multi-distribution learning using Turing machines. Suppose we want to say a Turing machine M \"solves\" multi-distribution learning. What are the inputs to M? Is it the collection of distributions $(D_1,...,D_k)$? Does the input also include the baseline class $\\mathcal{H}$? How are these inputs represented?\n\nIn addition, computational complexity is an asymptotic notion which requires a *sequence* of problems, not just a single instance. For example, discrepancy minimization instances are binary matrices $A \\in \\\\{0,1\\\\}^{n \\times n}$, which can be seen as a sequence of matrices indexed by $n \\in \\mathbb{N}$.\n\nIt might be helpful to first define a concrete sequence of multi-distribution learning problems that have natural bit representations. For example, one might define the domain ${\\mathcal X}_n = \\\\{0,1\\\\}^n$, the collection $(D_1,...,D_k)$ to be distributions whose marginal on $\\mathcal{X}_n$ is supported on the standard basis $\\\\{e_1,...,e_n\\\\}$, and the baseline class $\\mathcal{H}_n$ to be the set of halfspaces on $\\mathbb{R}^n$. Then, one can show that for each $n \\in \\mathbb{N}$, there is an efficient algorithm that maps a discrepancy minimization instance $A \\in \\\\{0,1\\\\}^{n \\times n}$ to a collection of measures (in bit representation) $(D_1,...,D_k)$ which are all efficiently computable and sampleable.\n\n**Motivation for deterministic classifiers?** While I also lean towards deterministic classifiers over randomized classifiers, are there well-founded reasons to prefer deterministic classifiers? It would be helpful if the authors could provide more motivation behind seeking deterministic classifiers.\n\n**Restriction to finite domains for Theorem 2.** In page 3, lines 96-98, the authors say,\n> *The restriction to finite domains $\\mathcal{X}$ seems to be mostly a technicality in our proofs. Since any realistic implementation of a learning algorithm requires an input representation that can be stored on a computer, we do not view this restriction as severe.*\n\nCould the authors elaborate on why the restriction to finite domains seems to be a mere proof technicality? What are some steps towards establishing the \"random rounding\" result for continuous domains? What assumptions (e.g., smoothness of the \"true\" regression function $f^*$) are needed?\n\nIn addition, I respectfully disagree with the second part of the quote. For example, sample complexity results in learning theory are independent of whether they can be implemented on a computer or not. Without further investigation, it seems rather premature to claim that the restriction to finite domains is not significant.\n\n**Storage reduction seems unlikely**. I am having trouble understanding how the storage requirement for the derandomized classifier could be reduced at all. Consider the following example. Let $\\mathcal{X} = \\\\{0,1\\\\}^n$, let the marginal distribution be the uniform distribution over $\\mathcal{X}$, and let $f^*(x) = 1/2$ for any $x \\in \\mathcal{X}$. Suppose the randomized classifier is a uniform mixture of two constant classifiers: all-ones $f_1(x) = 1$ for any $x \\in \\mathcal{X}$ and all-neg-ones, $f_{-1}(x) = -1$ for any $x \\in \\mathcal{X}$. Then, the random rounding algorithm outputs a purely random truth table indexed by $\\mathcal{X}$. However, random truth tables are incompressible.\n\nQUESTIONS:\n1. What is a formal definition of multi-distribution learning in the uniform computation model? Suppose we want to say a Turing machine M \"solves\" multi-distribution learning. What are the inputs to M? Is it the collection of distributions $(D_1,...,D_k)$? Does it also include the baseline class $\\mathcal{H}$? How are these inputs represented?\n\n2. Are there well-founded reasons to prefer deterministic classifiers over randomized ones?\n\n3. Could the authors elaborate on why the restriction to finite domains seems to be a mere proof technicality? What are some steps towards establishing the \"rounding\" result for continuous domains?\n\n4. In what sense can the storage requirement be reduced? It seems unlikely that *any* derandomized classifier can be compressed.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the setting of multi-distribution learning in the binary label setting, where the goal is to output a classifier that does well with respect to a set of distributions, rather than just a single distribution. In the agnostic case, all existing learning algorithms output a randomized classifier specified as a distribution over classifiers. The authors investigate to what degree this randomization is necessary, and show that in the most general case, de-randomizing may require super-polynomial training or evaluation time. On the positive side, they show that in the case where the set of distributions shares the same conditional label distribution for each covariate, they can give an algorithm for derandomizing a randomized classifier.\n\nSTRENGTHS:\n- The paper is written very clearly, and the proofs and statements are easy to follow, and seem to be correct. \n\n- The question of hardness of derandomization for multidistribution learning seems well motivated, and the hardness result requires minimal assumptions and is proved through an elegant reduction.\n\nWEAKNESSES:\n- It would be great to have some discussion of other related areas where a learning process outputs a randomized predictor that could be derandomized, such as in no-regret learning. \n\n- Similarly, the label-consistent setup seems equivalent to the problem of learning with covariate shift, and comparing the results to prior work in this area would be useful.\n\nQUESTIONS:\n- Is there any reason why the output is restricted to be a classifier rather than a deterministic predictor for the conditional distribution given x? Does allowing predictors to be outputted make the problem significantly easier?\n\n- Is there a reason why you provide sample complexity rather than runtime and evaluation time results for the label-consistent derandomization algorithm? This would be a better comparison to Theorem 1.\n\nLIMITATIONS:\nThe authors adequately address the assumptions they make and the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the *multi-distribution learning* setting of Haghtalab et al. (2022), where, given $k$ unknown data distributions $\\\\{\\\\mathcal{D}_1, \\\\dots, \\\\mathcal{D}_k\\\\}$, the goal is to find a classifier $f: \\\\mathcal{X} \\\\rightarrow \\\\{-1, 1\\\\}$ such that $f$ performs almost as well as $\\underset{h \\in \\mathcal{H}}{\\min} \\underset{i \\in [k]}{\\max} \\underset{{(x, y) \\sim \\mathcal{D}_i}}{\\mathrm{Pr}}[h(x) \\not = y]$ on every distribution $\\mathcal{D}_i$ for $i \\in [k]$.\n\nThe paper has two main results. The first is a negative result that states that, under mild conditions on $\\\\mathcal{H}$, any learning algorithm on $\\\\mathcal{H}$ that produces a classifier $f$ satisfying multi-distribution learning must have either super-polynomial training time or $f$ must have super-polynomial evaluation time. This is done through a reduction to the NP-Hard problem of Discrepancy Minimization. The second is a positive result that states that, under the condition of *label-consistency* (the conditional distributions of all the distributions agree, for all $x \\\\in \\\\mathcal{X}$) and finite input space, there does exist a learning algorithm that produces a deterministic classifier $f: \\\\mathcal{X} \\\\rightarrow \\\\{-1, 1\\\\}$ satisfying multi-distribution learning. It produces this classifier via black-box access to an existing multi-distribution learning algorithm that outputs a randomized classifier.\n\nSTRENGTHS:\nOverall, this paper is well-written and clear. I had no trouble following along with the arguments, and each proof is sketched out nicely before getting into the details. I believe the result does address an important issue in the multi-distribution learning literature, as I have personally found the fact that every existing multi-distribution learning relies on a randomized classifier predicting with a rather opaque distribution (usually obtained by running a no-regret algorithm) unsatisfying. \n\n**Originality:** The paper is original insofar as it provides the first (to my knowledge) derandomized classifier for the agnostic multi-distribution learning problem. I do wonder whether the algorithm itself can be made simpler, as it seems somewhat artificial to me to explicitly designate the outputs of our final classifier (defined as $\\hat{f}$) one-by-one through enumerating part of the input space $\\mathcal{X} \\setminus T$. That being said, taking the result as a purely a proof of existence, the work is novel and addresses an important question.\n\n**Quality:** The quality of the paper itself is good -- there are few typographical issues, the structure of the paper is clear, and the theorems are clearly presented. \n\n**Clarity:** There is no issue with clarity -- the authors have formatted the paper well, and the arguments are clearly presented.\n\n**Significance:** I believe the de-randomization question of multi-distribution learning to be significant, though more applied practitioners may view the paper as too theoretical. On the grounds that this is a learning theory work, however, this addresses an important question presented in COLT 2023 by Awasthi et al. (Open Problem: The Sample Complexity of Multi-Distribution Learning for VC Classes) through demonstrating a statistical-computational gap.\n\nWEAKNESSES:\nOne potential weakness in the work is that the final deterministic classifier relies on explicitly defining the outputs for all $x \\not \\in T$ by sampling from the existing multi-distribution learner's distribution. I don't find this to be too much of a weakness, as I do think that it is interesting that a deterministic algorithm *exists* in the first place, but I imagine that this might be viewed as a strong assumption. Perhaps the authors can provide some text motivating this deterministic classifier and its somewhat unwieldy final form (maybe this can be done by a brief discussion on the size of $T$?).\n\nAnother point that the authors could address a bit more is how natural the assumption of label-consistency for the positive result is. There is a brief reference to Ben-David and Urner (2014), but it would be helpful to include a further discussion on this assumption in the context of PAC learning. It seems that this is closely connected to the predominant assumption in the *covariate shift* literature, but a further discussion on perhaps the results of Ben-David and Urner (2014) in the body of this paper would be helpful to judge the strength of this main assumption. This is especially important because we are considering it in the context of multi-distribution learning.\n\nFinally, there are a couple of suggestions I had that may clarify the presentation of the paper:\n\n1. It might be a bit clearer to define $\\underset{\\mathrm{er}}{\\mathcal{P}}$ on a separate line (not in Equation 1). Personally, I took a while trying to find where $\\underset{\\mathrm{er}}{\\mathcal{P}}$ was defined in the paper, and it's an important quantity.\n2. In the Introduction, I would've appreciated some quick details on the existing algorithm for the deterministic classifier in the realizable setting. A one or two sentence sketch would be sufficient.\n3. It is unclear to me why the claim just above \"Our contributions\" about using the random classifier and Markov's inequality is the \"best we can guarantee.\" Can you justify that this is indeed \"the best\" argument for using a classifier drawn from the distribution $F$?\n4. I was initially confused about which definition you were taking for *label-consistent learning*, as you write, \"As a particular model of label-consistent learning, one may think of the deterministic labeling setup...\" so initially I thought you were restricting the assumption for the positive result to the deterministic labeling case with $f^*: \\mathcal{X} \\rightarrow \\{-1, 1\\}$. I would make clear (perhaps in a Definition environment) the label-consistent learning definition is, rather, the $\\mathcal{D}_i(y \\mid x) = \\mathcal{D}_j(y \\mid x)$ for all $i, j, x$ assumption.\n5. Small nitpick: In Section 2, you don't define the matrix $A \\in \\mathbb{R}^{n \\times n}$ before presenting the definition of Discrepancy Minimization.\n\nQUESTIONS:\nI had two major questions about the work: \n1. I was wondering about was whether it is possible to do such a de-randomization from scratch, i.e. without assuming black-box access to an existing multi-distribution learning algorithm. I wonder this because existing algorithms for multi-distribution, to my knowledge, all rely on applying Hedge (or a similar no-regret algorithm) to the $k$ distributions, which can be potentially expensive in situations with many distributions. I wonder if there are techniques to directly obtain a de-randomized classifier, potentially following the lines of the deterministic classifier for the realizable case.\n2. Did the authors consider any other assumptions besides label-consistency that might allow for a deterministic algorithm? \n\nThese last two questions aren't criticism on the work itself -- I was just curious if the authors thought about these issues.\n\nOne other thing that might be worth mentioning is that [Zhang et al. (2023)](https://arxiv.org/abs/2312.05134), Theorem 2 also presents a hardness result on derandomizing multi-distribution learning. However, their result is a statistical hardness result, while yours is computational. It would be helpful to put this paper in context perhaps with a couple of sentences in the Introduction.\n\nLIMITATIONS:\nThe authors have addressed the limitations of the work on a theoretical basis in the main text (see \"Discussion of Implications\"). Because this is mainly a theory paper, negative social impacts or broader impacts to applications seem minimal.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper studies the problem of multi-distribution learning in the agnostic setting. The authors provide a a computational lower bound for deterministic classifiers (based on some mild assumption on the hypothesis class, and the assumption $\text{BPP}\neq \text{NP}$), as well as an upper bound: an improper deterministic classifier for a special case of label-consistent distributions (i.e. when $D_i(y | x) = D_j(y | x)$ for all $i,j,x$). Their classifier uses a randomized learner as a black box, and its sample complexity is just almost the same as the sample complexity of the randomized learner.

STRENGTHS:
The results look interesting to me, they are non-trivial and novel (though I'm not an expert in the field, so I might have a wrong impression). The assumption of label-consistency looks natural to me, though it is not clear to me how restrictive it is.

WEAKNESSES:
While the results look new and non-trivial, I am not sure how strong the technical contribution is. I did not check the proofs in detail, and at the first glance the ideas don't look very sophisticated.

QUESTIONS:
As I already mentioned, it is not clear how restrictive label-consistency is. I'm not an expert in the area, and this assumption looks natural to me, but how is it related to standard assumptions? Was it already studied in prior works?

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-17