Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: On Margins and Generalisation for Voting Classifiers

In this paper, the authors take a generalization bound for stochastic majority votes from prior work and show how to derive a new margin bound for majority vote algorithms using Dirichlet priors. As stepping stone they prove a new result relating the misclassification loss to the stochastic voting loss. This bound is empirically verified to have better characteristics in that it comes closer to accurately predicting test set error than related work. The authors also derive an optimizable target version of their bound which appears to yield good results compared to other similar approaches in the literature on a variety of datasets.
strengths_and_weaknesses: This paper, although very dense, is making important contributions to the understanding of ensemble classifiers, in particular to the margin argument of why such classifiers generalize. It struck me as very insightful to be able to use the stochastic voting bound to prove the majority voting result. That the result also turns out to tighter than prior work as well as empirically perform well makes it very strong (and cool!). 
On the whole, I think this is a strong paper that is using very insightful connections from recent prior work to improve the state of the art generalization bounds on voting classifiers. 
questions: Two minor points: I would have liked to see a precise statement of Zantedeschi et al's bound in section 2.3. In particular, I missed the intuition behind "the use of such Dirichlet distributions...allows the correlation between voters to be more carefully considered." 
Second, I missed why in figure 2 the "our bound" results change for every baseline.
limitations: There is no specific limitations section, but the authors do describe gaps between their results and empirical results as well as cases when their bounds may not be accurate (albeit very briefly)
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper deals with the theory of (multi-category) pattern classification. The authors introduce a new PAC-Bayes bound for majority voting of finite ensembles of classifiers. It is based on Dirichlet posteriors. A variant is proposed to serve as the objective function of a training algorithm (train the model itself). Those two bounds are compared with the state-of-the-art ones in the framework of an empirical study.
strengths_and_weaknesses: The contribution appears to bring a significant improvement compared to the state of the art, even though it is not that original, since it is close to a result by Zantedeschi et al. (2021). I found it technically sound. It is noticeable that the paper is very well written (clear).
questions: A few typos should be corrected. Two examples :
-line 93 : complimentary -> complementary
-line 470 : pac-bayesian -> PAC-Bayesian
limitations: Not applicable
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper "On Margins and Generalisation for Voting Classifiers" proposes a novel generalization bound for voting classifiers (i.e. ensembles) using a Dirichlet posterior over the ensemble members. The novel bound explores a new direction for generalization bounds for ensembles and also seems to have decent results in practice. The paper is generally well-written and easy to follow. The theoretical contribution is clearly stated and the experimental evaluation seems decent. 
strengths_and_weaknesses: I like to paper and I do not have much to discuss about it to be honest. The paper combines different aspects from theory into a non-trivial, novel bound. It is well-written and easy to follow. The quality of the math seems sound, although I admit that I did only skimmed the appendix. I believe the use of the Dirichlet posterior is an interesting idea and can be used to as a starting point for future research. I have some minor remarks regarding the references that can easily be fixed for a camera ready and some question wrt. to the weights of each member in the ensemble (see below). In short: 

- (+) Novel and non-trivial combination of different results from literature
- (+) Generally well-written and comparably easy to follow
- (+) Decent empirical evaluation 
- (-) High test error in the empirical evaluation (see my question below)
- (-) The citations should be improved for more consistency. I suggest to double-check references {1,14,16,17,23,24,26,36,47} via https://dblp.uni-trier.de/ and copy/paste the extended Bibkeys
questions: 1) In the appendix you write that you initialize $\theta$ uniformly in $[0.01, 1]$ and then optimize over it. I assumed that one would initialize $\theta$ as the weight of each ensemble member for random forests, i.e. $\theta_i = 1/|\mathcal H|$ in the case of random forest. Why do you use different values here? Did I miss something?
2) The results in Fig 1. / 2. are comparably bad to e.g. classic RF. For example, a vanilla RF has a performance around $> 97 $% on mnist whereas you report errors in the range of $10-20 $%. I understand that raw performance is not the main focus of this paper (and hence I dont really care), but I am curios why there is this large performance gap. Do you think this is because $\theta$ is optimized afterwards?
3) I expected a generalization bound to be used as a "plug in" explanation for any voting classifier, e.g. as in Theorem 2. Now I wonder what the impact of $\theta$ is here. From my point of view, $\theta$ are simply the weights of the individual classifiers (e.g. equal for Random Forests or (normalized) floats for boosting). Is there any harm in using Theorem 2 to compute the error of, e.g. AdaBoost and if not, did you try that?
limitations: There might be some limitations wrt. to $\theta$, but I am not sure about this. Please see my question above.
review_title: null

Meta-review: metareview: All reviewers uniformly agree on the paper being interesting and worth publishing -- a very fine read. While the authors have already uploaded an updated version of their paper with minor revisions, I encourage them to use the camera-ready version to carry further improvements taking into accounts all reviews.


confidence: Certain
Decision: Accept