Title: Subsampled Ensemble Can Improve Generalization Tail Exponentially
Venue: NeurIPS 2025
Accepted: yes
Keywords: ensemble method

subsampling

heavy tail

exponential convergence

excess risk

generalization
Abstract: Ensemble learning is a popular technique to improve the accuracy of machine learning models. It traditionally hinges on the rationale that aggregating multiple weak models can lead to better models with lower variance and hence higher stability, especially for discontinuous base learners. In this paper, we provide a new perspective on ensembling. By selecting the most frequently generated model from the base learner when repeatedly applied to subsamples, we can attain exponentially decaying tails for the excess risk, even if the base learner suffers from slow (i.e., polynomial) decay rates. This tail enhancement power of ensembling applies to base learners that have reasonable predictive power to begin with and is stronger than variance reduction in the sense of exhibiting rate improvement. We demonstrate how our ensemble methods can substantially improve out-of-sample performances in a range of numerical examples involving heavy-tailed data or intrinsically slow rates.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the technique of ensemble learning, a popular paradigm consisting in aggregating several weak learners into a strong model (e.g., combining decision trees into tree ensembles). The authors propose the novel methodology of simply selecting the best base learner out of the pool of models trained on different subsamples of the data. In the easiest case, this happens via majority voting. They show that this improves the generalisation bound of the model, guaranteeing an exponentially decaying generalisation tail even where the general problem has a slow polynomial decay.\n\nMore specifically, in a discrete problem the model appearing more frequently is selected as the output model (MoVE). In continuous problems, this degenerates as all learned models likely appear exactly once. Therefore, ROVE chooses the model that has the highest likelihood of being eps-optimal.\n\nThe authors then formally derive theoretical bounds for the exponentially decaying generalisation tail in both scenarios. \n\nThey then experiment with MoVE and ROVE, which actually comes in two different versions, one where all data is used in both steps, one that splits the data between learning candidate models and performing eps-optimality vote. Experiments are performed on synthetic regression datasets tackled with MLPs, and on four discrete heavy-tailed stochastic programs. For the regression cases the proposed algorithms seem to work better than a single base learner. A further comparison is drawn with bagging - here we see that bagging wins on light tailed, and viceversa, which is reasonable. Performances on the stochastic programs look competitive, and ROVE seems the best alternative.\n\nSTRENGTHS AND WEAKNESSES:\nSTRENGTHS\n\n* significance: ensembling is a very popular technique and studies advancing its applicability are definitely valid to the ML and optimisation communities\n\n* quality: the paper is well written, has a solid theoretical justification and an extended experimental evaluation (I have only one concern that I express in my questions)\n\n* originality: the work is original and seemingly valid for any kind of ensemble - therefore it could lead to practical improvements in diverse areas where current ensembling solutions are not satisfying enough\n\nWEAKNESSES\n\n* the work is quite theoretical and I do not necessarily see an immediate reflection in ML models, e.g., I am still a bit unsure that a single tree can do better than an ensemble in a lot of scenarios. See my related question. \n\n * while I get that the main point is a higher level contribution, it feels a bit weird to me that a paper on ensembles does not have any experiment related to tree ensembles, arguably the most popular ensembling technique in ML - see my related question later\n\nQUESTIONS:\n* I'm not entirely sure I get the discrete scenario. The majority vote in MoVE happens over the different trained models. Does that mean multiple models that are exactly the same are the output of the training on different subsamples? e.g., we would need to have two identical decision trees? Or is this for example related to the numerical solution of a stochastic program? Could you please provide a practical example of a discrete scenario? It's probably just a lack of expertise on my side.\n\n* In addition to the formula in the algorithm, could you better explain in a couple of sentences and/or with an example what it means that a model is eps-optimal?\n\n* The proposed method envisions always using one single learner (extracted through majority voting or finding the eps-optimal one), and justifies how this improves generalisation. Intuitively, often using boosting/bagging is a much more powerful solution (e.g. boosted ensembles such as XGB perform way better than single decision trees). Indeed in Figure 2 you show that for light-tailed distributions subagging wins over ROVE. \n     - Is this always the case? Does your method always wins for heavy-tailed distributions?\n     - Were you able to find a \"cutoff\" which tells when your method is preferable, depending on how data is distributed? Can you extract this somehow from the training data?\n    - Have you compared your algorithms to other ensembling techniques other than bagging (from Figure 2)?\n\n* Arguably the most popular type of ensembles in machine learning is tree ensembles. I wonder why no experiments were presented using decision trees? Do you know if your algorithm performs well on those?\n\nLIMITATIONS:\nThe authors address one potential limitation in the last paragraph. The work has no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new method for aggregating base learners, each trained on a subsample of the training set. They prove and show for toy examples that the tail probability for the corresponding excess risk can be exponentially small, when the assumptions are met. The empirical performance of the proposed method is also demonstrated.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The proposed method is reasonable and potentially useful when the tail probability needs to be controlled. The results presented in Figs. 1 and 2 are consistent with intuitive expectations.\n\nWeaknesses: \n1) Theoretically, although the authors claim that the performance enhancement of their method is ``agnostic to the underlying base learner,'' the crucial condition that $\\eta_{k,\\delta}>0$ seems hard to verify for a given algorithm $\\mathcal{A}$ and a given $\\delta>0$. If such a condition only holds for extremely large $\\delta$ while the population risk is bounded, then such a result may not be very informative. This scenario is possible. Consider the case of a finite $\\Theta$ for simplicity. Suppose that, for some dataset, the algorithm $\\mathcal{A}$ outputs $\\theta\\in\\Theta$ with roughly the same probability, and for the population risk minimizer $\\theta^\\ast$, we have that $L(\\theta^\\ast)\\ll L(\\theta)$ for all $\\theta\\neq \\theta^\\ast$. Then $p_k^{\\max}\\approx 1/|\\Theta|$, while $\\mathcal{E} \\approx 1 - 1/|\\Theta|$ for a large range of $\\delta$. Then the condition that $\\eta_{k,\\delta}>0$ cannot be satisfied in such a case. (Here $\\mathcal{E}$ denotes $\\mathcal{E}_{k,\\delta}$. There is some unknown reason that this cannot be displayed correctly.) \n2) Also, the condition $\\eta_{k,\\delta}>0$ is unreasonably strong for me. On the one hand, $p_k^\\{\\max}$ is the probability that corresponds to the mode in $\\Theta$. On the other hand, $\\mathcal{E}$ is the tail probability for the excess risk incurred by some random $\\theta$. There is no reason that these two can be compared. By requiring the former be (strictly) greater than the latter, it is qualitatively requiring that the excess risk for the ``mode'' base learner can almost be controlled, which leads to the exponential tail. In my opinion, this is too strong. It is also unsatisfactory that there is no explicit dependence on $\\delta$ in the final expression of the tail bound.  \n3) There are some technical details that might not be totally correct, or need further clarification. (See Questions below.)\n4) Practically, the authors only compare their method with the conventional bagging in terms of the mean test error, rather than the tail probability. I believe it makes much more sense to compare the tail probability with bagging.\n\nQUESTIONS:\nQ1. If $\\eta_{k,\\delta}\\leq0$, what can be said?\n\nQ2. What are the reasonable conditions under which $\\eta_{k,\\delta}>0$?\n\nQ3. In Figs. 1 and 2, can you add the tail probability for bagging?\n\nQ4. In Example 1.1 and/or the linear regression example, can you provide the corresponding tail bound for bagging? Is there any substantial improvement by applying your proposed method?\n\nQ5. In Algorithm 2, can you speficy ``Choose $\\epsilon\\geq 0$ using the data...''?  How is $\\epsilon$ picked practically such that it satisfies the conditions in Theorem C. 11, which seem hard to check? \n\nQ6. In Theorem C.11, it is unclear to me in which ways $k_1,k_2,B_1,B_2$ go to infinity as $n$ goes to infinity, respectively. It seems problematic in some cases. Note that the tail probability on the right-hand side of Eq. (34) is greater than $B_1e^{-B_2/C_3}$. If $B_2=\\Theta(\\log n)$ and $B_1=\\Theta(n^{1+0.01})$, then the bound is trivially going to infinity. How can the left-hand side converge to 0?\n\nQ7. This is a minor comment. In Technical Novelty, the authors mention ``a sharper concentration result for U-statistics with binary kernels, improving upon standard Bernstein-type inequalities.'' I believe the KL divergence type concentration inequality is not new. In D.2, once it comes to the step that $\\tilde \\kappa$ can be viewed as a binomial random variable, the rest is known. The novelty of the work might be the application of this result in U-statistics with binary kernels.\n\nLIMITATIONS:\nYes. (I do not foresee any negative societal impact of the work.)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel ensemble learning method comprising MoVE (Majority Vote Ensembling) and ROVE (Retrieval and $\\epsilon$-Optimality Vote Ensembling), which leverages subsampling and model selection strategies to achieve exponential decay in excess risk, claiming to significantly outperforming traditional ensembling techniques which have polynomial decay, particularly in heavy-tailed data distributions. By training multiple base learners on data subsets and combining them through majority voting or likelihood-based retrieval, the approach ensures stronger generalization even when individual learners exhibit only polynomial decay in excess risk. Theoretical analysis confirms the presence of exponential tails in the generalization error. Experiments were conducted on: the MLP regression problem with synthetic data; four discrete stochastic programs: resource allocation, supply chain network design, maximum weight matching, and stochastic linear programming; and one continuous: mean-variance portfolio optimization, showing improved results.\n\nSTRENGTHS AND WEAKNESSES:\nOne-line summary: the paper is strong in theoretical contributions but, at least to me, not totally convinced in terms of practicality. My initial rating is based on my assessment that the theoretical contributions slightly outweigh the practical concerns.\n\n## Strengths\n\n1. This is a paper that is very solid in terms of theoretical contributions but well-written in such a way that the audience can manage to follow. Example 1.1 is particular useful in terms of motivating what the paper mainly contributes: even if base learners have polynomial decay in excess risk, classical bagging approach would result in polynomial decay but with the paper's proposed approaches, one can obtain exponential decay.\n\n2. Apart from the main theoretical contribution, which is exponential decay instead of polynomial decay, there are a number of small theoretical contributions in deriving the main contribution: a sharper concentration result for U-statistics with binary kernels improving upon Bernstein-type inequalities, analysing regret sensitivity, and developing a uniform law of large numbers (LLN) for $\\epsilon$-optimal events. \n\n3. Experimental results support the theoretical claims.\n\n## Weaknesses\n\n1. Although the main claim is about going from polynomial decay to exponential decay for the excess risk, I do not think this is too big of a claim because it is well known that in classification we have exponential decay. I maybe wrong but perhaps in regression the best bounds have not approached exponential decay yet and this paper fills in that missing link.\n\n2. In practice, when it comes to ensemble learning these days it is very popular to use gradient boosting (e.g. XGBoost or LightGBM for implementations) for both regression and classification problems as it has shown successes in many benchmarks. Cited in the paper but gradient boosting is nowhere to be found in the experiments, which is a big surprise to me. I think gradient boosting should have served as a strong baseline to compare against, alongside any bagging approach. Why? Because problem-wise, these approaches solve the same problem. In contrast, bagging and even advanced bagging approaches like random forests, are no longer very popular, as they are not as performant as gradient boosting. Gradient boosting approaches have a difference, maybe strength (which I may be wrong), over the proposed MoVE/RoVE/RoVEs approaches here in that they build new base learners based on the performances of previous base learners, whereas the approaches here build all base learners at once initially. Although the theory side may be lacking, intuitively such strategy that gradient boosting adopts may lead to a faster exponential decay rate.\n\n3. Although the idea of majority voting sounds interesting at first, it seems to come with more drawbacks than benefits. In order to do majority voting, you have to discretize the parameter space $\\Theta$. In this deep learning era where model weights are in the order of billions, discretizing the parameter space does not sound practical. Initially MoVE only works with discrete $\\Theta$. The authors have come up with the $\\epsilon$-optimal notion and derived RoVE/RoVEs to work with continuous $\\Theta$, which appears to be an attempt to \"cluster\" the continuous $\\Theta$ into a countable set of models that the algorithms can manage. However, when accuracy becomes highly demanded, one has to bring down $\\epsilon$ very close to 0. But that might explode exponentially the number of discrete models that the algorithms have to train. Wouldn't that mean $B$, the ensemble size, also has to be increased exponentially? The proposed theory treats $B$ as a constant (relying on $n/k$ only to be more precise), which, I think, has a tendency to hide the complete view.  $B$ deserves to have deeper analysis, especially when $\\epsilon$ becomes very close to 0.\n\nQUESTIONS:\n1. As claimed by the authors, classification is not addressed here due to the cross-entropy classification loss. It is somewhat reasonable but am not entirely convinced. Heavy tailed distribution can show up when the number of classes becomes very large, e.g. thousands, and we change our view point to the distribution of the classes. In such case it would be very interesting to see how the proposed approaches here perform. Do you have any comment on this point?\n\n2. I have a suggestion in mixing the approaches here with gradient boosting, which may be worth exploring. You can use MoVE/RoVE/RoVEs as a base learner at each iteration of gradient boosting, limiting to a small, managable size $B$ at each iteration. In such a way, you have a strong base learner than decision stumps that gradient boosting approaches are using. It would be interesting to see if such a mixed approach could lead to a result that is better than both vanilla gradient boosting and MoVE/RoVE/RoVEs.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a new ensemble learning approach that leverages subsampling to dramatically improve the tail behavior of generalization error. In contrast to classical bagging (which reduces variance by averaging outputs of multiple models), the authors propose to aggregate at the parameter/model level rather than just the output level . Specifically, they train many models on random subsamples of the data and then select a final model via a majority-vote scheme across those subsamples. The main theoretical result is that this procedure yields an exponentially decaying tail for the excess risk (generalization error)\n– even when each individual “base” learner only achieves a polynomial (slow) tail decay . In other words, with high probability the ensemble’s error is as small as the best model’s, providing a form of oraclelevel performance guarantee. This improvement is agnostic to the base learning algorithm: it holds for standard empirical risk minimization as well as more robust schemes (e.g. distributionally robust optimization or regularized methods) . The paper introduces a family of methods (named MoVE, ROVE, and ROVEs) to implement this idea, and demonstrates in experiments that these ensemble methods significantly improve out-of-sample performance on tasks with heavy-tailed data or inherently slow convergence rates. The authors also provide open-source code for their ensemble algorithms, supporting reproducibility.\n\nSTRENGTHS AND WEAKNESSES:\nStrength : The paper presents a new ensemble learning framework that significantly enhances theoretical approaches for learning with heavy-tailed data distributions, a complex scenario frequently encountered in safety-critical fields like finance, medicine, and systems influenced by noisy or adversarial inputs. The authors show that the suggested ensemble method can turn polynomially decaying generalization tails into exponentially decaying ones. This is a big step forward in learning guarantees, not just a small one. This is especially important for situations when high-confidence reliability is very important. The method is also base-learner agnostic, which means it might be a powerful and versatile \"wrapper\" that makes a number of existing learning algorithms more resilient. The work is well-organized and written in a formal academic language, with well defined terms (such \"excess risk\" and \"tail decay rate\") and descriptive names for the approaches (MoVE, ROVE, ROVEs) that make it apparent what they do. Each approach is also put in context by explaining what it's for, and the theoretical ideas are backed up with clear explanations and strong visuals, such figures and examples, that help make sense of how the methods work when there are heavy-tailed situations.\n\nWeaknesses : The paper has some good points, but it also has some problems when it comes to being easy to read, useful in real life, and useful in a wider sense. The main idea behind this method is to choose a model by majority vote, however the way it is presented could be misleading. Many readers could think of \"majority voting\" in the traditional sense (i.e., voting on predictions) instead of voting on the models themselves. This unusual framing, which doesn't include an early, step-by-step pseudocode or algorithm description, makes it harder to understand. Also, important details like how to choose the number of subsampled models ($B$) or the best subsample sizes are not stressed in the main text and may be hidden in the appendix. This makes it tougher for readers to use or duplicate the strategy in real life. The method's practical applicability may be confined to specific contexts, with its benefits being most evident in situations characterized by heavy-tailed loss distributions or suboptimal convergence behavior. In typical ML applications characterized by sub-Gaussian noise or well-behaved distributions, the ensemble may provide minimal to no advantages compared to simpler bagging or robust loss techniques. Also, the predicted exponential improvements can be asymptotic, which means that we don't know if we can see real advantages at realistic sample sizes (like $n = 10^4$). In general, the method looks good on paper, but it might not be used in real-world ML pipelines until it is tested further, made simpler, and given clearer instructions for how to use it.\n\nQUESTIONS:\n1. Clarification of the \"Majority Vote\" Process: Could you likely explain how the model selection process works in more specific terms, maybe by using a simple algorithmic pseudo-code? This statement was a little perplexing. For example, if the model parameter space is continuous, how does a \"vote\" work? It would help readers and practitioners use the procedure correctly if it were made clear whether this means picking the model that does best on most subsample runs and how ties or near-ties are handled. What criteria on the model class cause several subsamples to converge to the same \"majority\" model?\n\n2. Assumptions for guarantees of exponential tails: The idea says that the tails of excess risk will decrease exponentially. What assumptions about the data or the base learner must be true for this assurance to hold? For example, do we need a finite second moment or only a moment condition on the loss distribution? If the data has a very heavy tail (for example, infinite variance), does the ensemble still get an exponential tail, or would the rate improvement be limited? It would be beneficial for the paper to clearly describe the criteria (e.g., “the loss belongs to class $\\mathcal{H}_\\alpha$” or analogous statements) under which Theorem 1 (or the principal conclusion) is valid. This tells readers when it's okay to use the method.\n\n3. Cost and scalability of computing: The ensemble trains $B$ models on subsamples. What advice can you give on how to choose $B$ (the size of the ensemble) and the subsample percentage in real life? The trials probably employ some defaults, but it would be helpful if the paper talked about the trade-offs (more models for greater tail assurance vs. higher computation). Also, have you tried the method on bigger problems, such deep neural networks or big datasets? If not, do you think there will be any problems (like problems with optimization or too much variation in the outcomes of subsamples) when you scale up? It would be helpful to include some examples or explanations of how performance becomes better as $B$ gets bigger.\n\n4. Comparison to Other Robust Methods: How does the subsampled ensemble stack up against other robust methods for heavy tails, including applying a robust loss function (like Tukey's loss or trimmed mean aggregation) or the median-of-means applied directly to outcomes? Kwon et al. (2021) specifically suggested a robust model selection utilizing MOM—how does your methodology vary or exhibit greater generality? A clear explanation of how this method is different from median-of-means or bootstrap-of-means methods would make the newness clearer. For example, can a basic median-of-means estimator on the losses fail to produce exponential tails while your model-selection method works? Such information would help readers know when to choose the suggested ensemble over other strong estimators.\n\n5. Things to think about in the real world and examples of how to use it: The paper centers on heavy-tailed synthetic or regulated trials. Can you give me any instances of real-world datasets or areas where you think base learners will only have polynomial tail decay and will benefit a lot from your method? For example, would this be useful for reinforcement learning (where returns can be heavy-tailed), financial risk models, or any other big benchmarks? It would also be helpful to talk about the ensemble's failure modes, such as when subsampled voting might impair performance or not produce an exponential tail (for example, if the subsamples are too small to train usable models or if the data aren't heavy-tailed at all). This kind of advice would help practitioners who are thinking about using this ensemble technique make their work more useful.\n\nLIMITATIONS:\nThe main problem with this task is that it takes a lot of computer power. To get the exponential tail bound, you need to train and test a lot of models on different data sets, which might be costly. This could make training take a lot longer in situations like deep learning than it would with just one model. Another problem is that the theory only looks at the worst-case tails of the error distribution. When the ensemble tries to get the best performance in the worst-case scenario (high-confidence bounds), it might give up some average-case accuracy or introduce bias. For example, it might choose a model that is safer but not quite as good on average. Users should know that the strategy works best when there are genuine concerns of heavy tails; otherwise, a smaller ensemble or single model can be enough. There is also a statistical limitation: the method implies that we can accurately find the \"best\" model by looking at how well it does on a subsample. If the number of subsamples $B$ is too small or if the performance of the subsamples is sufficiently noisy, most of vote could choose a model that isn't good enough. In rare circumstances, as when the subsamples are very small or the model classes are very complicated, the ensemble could not work better and could potentially make things worse because the voting results are unstable. These subtleties imply that the method necessitates meticulous adjustment of $B$ and subsample size, and perhaps requires validation using a hold-out set, if feasible.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}