Title: Sample-Adaptivity Tradeoff in On-Demand Sampling
Venue: NeurIPS 2025
Accepted: yes
Keywords: On-demand sampling

multi-distribution learning

adaptivity
Abstract: We study the tradeoff between sample complexity and round complexity in *on-demand sampling*, where the learning algorithm adaptively samples from $k$ distributions over a limited number of rounds. In the realizable setting of Multi-Distribution Learning (MDL), we show that the optimal sample complexity of an $r$-round algorithm scales approximately as $dk^{\Theta(1/r)} / \epsilon$. For the general agnostic case, we present an algorithm that achieves near-optimal sample complexity of $\widetilde O((d + k) / \epsilon^2)$ within $\widetilde O(\sqrt{k})$ rounds. Of independent interest, we introduce a new framework, Optimization via On-Demand Sampling (OODS), which abstracts the sample-adaptivity tradeoff and captures most existing MDL algorithms. We establish nearly tight bounds on the round complexity in the OODS setting. The upper bounds directly yield the $\widetilde O(\sqrt{k})$-round algorithm for agnostic MDL, while the lower bounds imply that achieving sub-polynomial round complexity would require fundamentally new techniques that bypass the inherent hardness of OODS.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of adaptive data collection for multi-distribution binary classification. Multi-distribution refers to the learner being evaluated on the error for data coming from across $k$ distributions. The adaptive data collection is formalised by a procedure over $r$ rounds where in each round the learner can choose the number of samples to collect from each of the distributions (and this can be based on samples collected in previous rounds). The aim of the paper is to understand the trade-off between $r$ and the total number of samples needed for $\\epsilon$ accuracy (on the multi-distribution error in a PAC sense). In the realizable case (where a perfect predictor exists), they provide upper and lower bounds that characterise the (near) optimal trade-off. In particular, O(log k) adaptive rounds is optimal to achieve the optimal sample complexity. In the more challenging agnostic case (where a perfect predictor does not exist), an algorithm is provided that achieves the near-optimal sample-complexity in a number of rounds that is independent of $\\epsilon$. The optimization via on-demand sampling framework is introduced whose connection to the multi-distribution problem allows to motivate the difficulties in improving the round-complexity of the agnostic case, as well as establish general results on the trade-off between sample complexity and adaptivity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is generally well-written and easy to follow (except a few parts – see weaknesses). The algorithms are well explained and the proof sketches provided in the main text are nice and provide some useful intuitions.\n2. Realizable case: An upper-bound is established that captures a trade-off between adaptivity and sample-complexity. In particular, I believe the main point that can be concluded from the upper and lower bound is about the optimality (sufficiency + necessity) of $\\log k$ rounds to achieve the optimal sample-complexity, completing the characterization of optimality for adaptive sample-complexity.\n3. Agnostic case: An upper-bound is established that achieves the near-optimal sample-complexity with a round-complexity that is independent of the accuracy $\\epsilon$, improving on prior work whose algorithms had round-complexity that depended polynomially on $\\epsilon$.\n4. The algorithms are non-trivial extensions of algorithms from prior work.\n\nWeaknesses:\n1. Some of the technical description is a bit unclear. There seems to be some missing notation in the description of Algorithm 1, e.g. $q_t^m$ is not defined (I guess this means m samples from $q_t$ ?), $q_{t+1}$ / $q_t$ is not defined (only $q_{t+1}(j)$ is). Additionally, it is not clear where $p$ and $\\varepsilon_A$ are used in the algorithm, which then makes the following discussion of the technical overview a bit unclear. In particular, my understanding is that the learner $A$ is called to to learn a predictor $q_t$ that minimizes the fraction of distributions (as weighted by $q_t$) where the weights $q_t(j)$ have been adjusted in previous rounds according to the errors being more or less than $\\tau/2$ but I don’t see how the equation below l.142 fits in.\nThe proof sketch of Theorem 2 for $r=2$ provides some nice intuition but the explanation of the way $diff_i$ as uniformly random permutations is not clear. It is not clear to me what is meant by a uniformly random permutation of a copy of $\\Theta(d)$. Is $diff_1$ a sample from a permutation of $[1,…,d]$ ? Is $diff_1$ itself a permutation, though my understanding is that $diff_i$ is the dimension of subspaces that more or less partition the space, I.e.\\ an integer. Also, if possible I think it would be clearer to avoid using $\\Theta$ notation here and explicitly put the values. \n2. The lower-bound in Theorem 2 has a $k^{1/r}$ dependence while the upper bound is $k^{2/r}$. However, this is enough to establish the necessity of the $\\log k$ rounds, which in my understanding is the main contribution this result allows to establish for the realizable case (and perhaps should be stressed more ?), especially since known algorithms with optimal sample-complexity already had a round complexity of $\\log k$. Indeed, for the case where $r$ is a small constant, the upper-bound from Theorem 1 is an interesting result, but does not match the lower-bound.\n3. The high-level connection between optimization via on-demand sampling and MDL is discussed in lines 260-265 (and again in Section 5.3), however it is unclear how the results from each section are connected. What is the connection between Theorem 4 and proposition 1 ? And with this absence of connection, it is a bit unclear how the results from Section 5 fit in with the rest of the paper. In particular, I understand how Theorem 9 provides some insight into how well lazy-hedge can perform for MDL but it is unclear how Theorem 4 does this from an upper-bound perspective. More generally, while Section 3 has a clear structure that is easy to follow, it becomes a bit harder to follow the narrative through Sections 4 and 5. Also, in Section 4 a bit more discussion could be provided as currently the results are simply stated without much discussion.\n4. Proposition 1 provides an algorithm with near-optimal sample-complexity with a poly(k) round-complexity but no lower-bound is provided.\n\nQUESTIONS:\n1. In the discussion following Theorem 1, I am not sure about the claim at l.130 that $r = \\log k$ is the “most sample-efficient”. Given the upper-bound in Theorem 1, the most sample efficient is when $r \\rightarrow \\infty$ - I guess what was meant is that $r = \\log k$ already allows to recover the order of the optimal sample complexity. But really if we ignore what we know to be the optimal sample-complexity, the bound in Theorem 1 suggests that we should set $r$ to match the second term of the bound. And at a higher level, what it says is that the adaptivity level (the value of $r$) should be set as a function of k and d and in particular will behave somewhat differently when k>d or d>k. Have you thought about this ? And also how this might be relevant to settings where $k$ may be unknown, how hard would it be to extend these results ?\n2. In l.132, when $r = 5$ is chosen in Theorem 1, why is it $\\sqrt{k}$ in the sample complexity and not $k^{2/5}$, is this a typo ? I think there is also a typo l.157: it should be $\\sqrt{k}d$ not $\\sqrt{d}k$, right?\n3. In weakness 4., it was mentioned that no lower-bound is provided for the round-complexity in the agnostic case. In section 5.3, it was discussed the improving the round-complexity required a different approach. Could you comment on whether you believe the $\\sqrt{k}$ round-complexity could be improved and if so what the optimal round-complexity could be?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-18"}

comments: SUMMARY:
This paper works on reducing the round complexity for multi-distributional learning problem. Multi-distribution learning (MDL) focuses on learning multiple distributions with on-demand sampling. Recent work established near-optimal sample complexity for MDL. In this work, the authors try to reduce the round complexity with lazy updates. They provide a trade-off between the round complexity and the sample complexity for the realizable case, and prove a sub-linear round complexity for the agnostic case.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The major contribution of this work is to establish the trade-off between the adaptivity and the sample complexity. This represents a valuable contribution to related fields. 
2. In technique, the major contribution is  a novel application of a variant of the AdaBoost algorithm with a particular notion of margin.  The construct of the lower bound is also a significant technical contribution.

Weakness:
1. The results in the agnostic setting are relatively weak due to the absence of statistical lower bounds.

QUESTIONS:
1. Can you provide some intuition for why the lower bound is not polynomial in $\epsilon$?  Is there any method to combine your arguments with the classical $\tilde{\Omega}( \frac{d+k}{\epsilon^2})$ lower bound (without limitation on round complexity) to obtain a better rate?
2. I wonder whether there is substantial difficulty in extending the upper bounds for realizable setting to the agnostic setting. In particular, fix $r\leq \sqrt{k}$, could you prove a sample complexity upper bound better than the trivial bound $dk/\epsilon^2$ for the agnostic setting?

LIMITATIONS:
The theoretical results only apply to the specific parameter regimes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies an adaptive learning process in the Multi-Distribution Learning (MDL) setting, and characterizes a tradeoff between the sample complexity and round complexity. Specifically, it studies to what extent an adaptive sample collection can achieve a near-optimal sample complexity while maintaining a beneficial round complexity. Under the realizable setting, it characterizes the sample complexity as $dk^{\\Theta(1/r)}/\\epsilon$, where $r$ is the round complexity, using a variant of the classical AdaBoost algorithm. In the agnostic case, it shows that with $\\tilde{O}(\\sqrt{k})$ rounds, a sample complexity of $\\tilde{O}((d+k)/\\epsilon^2)$ can be achieved through an revised algorithm from that of [ZZC+24] (by allowing laze sampling with a cap). From a theoretical perspective, the paper also introduces the framework of Optimization via On-Demand Sampling that naturally connects to the multi-distributional learning problems, which implies both the upper and lower bounds for the round complexity with specified sample overhead.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper considers an important problem of multi-distributional learning under the adaptive setting, and provides a theoretical foundation for studying the tradeoff between the sample and round complexity. It studies the problem in both realizable and agnostic settings, providing both upper (through well-designed algorithms) and lower bounds (through rigorous analysis). The paper is very well presented with original and insightful discussions. The work is very influential in adaptive learning settings, and could benefit in this area of research, including on-demanding sampling, multi-arm bandit problems, continual learning, to name a few.\n\nQUESTIONS:\nDid authors consider other boosting algorithms that might further reduce the sample complexity?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies on-demand sampling, where there are $K$ distributions and the goal of Learner is to learn a hypothesis that minimizes the worst-case error. They consider two classical settings in statistical learning: the realizable setting and the agnostic setting. They present both the upper and lower bounds results. For this work, they study the fundamental trade-off between sample complexity and round complexity, which is very important.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: the studied trade-off is very fundamental. \n\nWeakness: for some part about the presented Algorithm 1, it is not clear. The details are in the box below.\n\nQUESTIONS:\n1. What is adaptive sampling, or how to decide in each round, whether it is adaptive sampling or non-adaptive sampling? And what is full adaptivity and full non-adaptivity? I hope to see more background knowledge of adaptive sampling.\n\n2. Is my understanding right the provided results in the realizable setting can only improve the existing non-adaptive sampling results? From the content below Theorem 1, it seems it cannot improve the results shown in [BHPQ17, CZZ18, NZ18].\n\n3. Can you give more explanations and intuitions about the agnostic results such as how to get rid of the $\\epsilon$ in your new results? And why the round complexity does not need to be dependent on $\\epsilon$? For me, the existing results make more sense.\n\n4. For Algorithm 1, how to decide the number of samples that are distributed according to each of the K distribution in each round? Can it be varied among distributions?\n\n5. The discussion below Theorem 1, it will make it look nicer if, the two special cases, the case where $r = \\log k$ and  $r = O(1)$, are presented in Table 1.\n\n6. Also, for the discussion below Theorem 1, it seems that when $r = \\log K$, it cannot recover the existing results.  Why claim it is almost recovered? I think it only improves the non-adaptive sampling results. \n\n7. Algorithm 1 inputs round $r$, can it be chosen adaptively, for example, based on the past collected data? \n\n8. Maybe I missed something in  Algorithm~1. what are $q_t^m$ and $D_j^n$  ? \n\n9. In algorithm 1, in step 3, it seems we do not know $D_i$ right? So, how to compute $q_1$?\n\n10. For Theorem 2, it is a minimax lower bound or a lower bound for this specific algorithm? If it is a minimax lower bound, it seems to be contradicted with the existing results in [BHPQ17, CZZ18, NZ18] in the realizable setting.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-10"}