Learning Stochastic Majority Votes by Minimizing a PAC-Bayes Generalization Bound

1 Introduction . By combining the outcomes of several predictors , ensemble methods [ Dietterich , 2000 ] have been shown to provide models that are more accurate and more robust than each predictor taken singularly . The key to their success lies in harnessing the diversity of the set of predictors [ Kuncheva , 2004 ] . Among ensemble methods , weighted Majority Votes ( MV ) classifiers assign a score to each base classifier ( a.k.a . voter ) and output the most common prediction , given by the weighted majority . When voters have known probabilities of making an error and make independent predictions , the optimal weighting is given by the so-called Naive Bayes rule [ Berend and Kontorovich , 2015 ] . However , in most situations these assumptions are not satisfied , giving rise to the need for techniques that estimate the optimal combination of voter predictions from the data . Among them , PAC-Bayesian based methods are well-grounded approaches for optimizing the voter weighting . Indeed , PAC-Bayes theory ( introduced by Shawe-Taylor and Williamson , 1997 , McAllester , 1999 – we refer to Guedj , 2019 for a recent survey and references therein ) provides not only bounds on the true error of a MV through Probably Approximately Correct ( PAC ) generalization bounds ( see e.g . Catoni [ 2007 ] , Seeger [ 2002 ] , Maurer [ 2004 ] , Langford and Shawe-Taylor [ 2002 ] , Germain et al . [ 2015 ] ) , but is also suited to derive theoretically grounded learning algorithms ( see e.g . Germain et al . [ 2009 ] , Roy et al . [ 2011 ] , Parrado-Hernández et al . [ 2012 ] , Alquier et al . [ 2016 ] ) . Contrary to the most classical PAC bounds [ Valiant , 1984 ] , as VC-dimension [ Vapnik , 2000 ] or Rademacher-based bounds [ Mohri et al. , 2012 ] , PAC-Bayesian guarantees do not stand for all hypotheses ( i.e . are not expressed as a worst-case analysis ) but stand in expectation over the hypothesis set . They involve a hypothesis space ( formed by the base predictors ) , a prior distribution on it ( i.e . an a priori weighting ) and a posterior distribution ( i.e . an a posteriori weighting ) evaluated on a learning sample . The prior brings some prior knowledge on the combination of predictors , and the posterior 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . distribution is learned ( adjusted ) to lead to good generalization guarantees ; the deviation between the prior and the posterior distributions plays a role in generalization guarantee and is usually captured by the Kullback-Leibler ( KL ) divergence . In their essence , PAC-Bayesian results do not bound directly the risk of the deterministic MV , but bound the expected risk of one ( or several ) base voters randomly drawn according to the weight distribution of the MV [ Langford and Shawe-Taylor , 2002 , Lacasse et al. , 2006 , 2010 , Germain et al. , 2015 , Masegosa et al. , 2020 ] . This randomization scheme leads to upper bounds on the true risk of the MV that are then used as a proxy to derive PAC-Bayesian generalization bounds . However , the obtained risk certificates are generally not tight , as they depend on irreducible constant factors , and when optimized they can lead to sub-optimal weightings . Indeed , by considering a random subset of base predictors , state-of-the-art methods do not fully leverage the diversity of the whole set of voters . This is especially a problem when the voters are weak , and learning to combine their predictions is critical for good performance . Our contributions . In this paper , we propose a new randomization scheme . We consider the voter weighting associated to a MV as a realization of a distribution of voter weightings . More precisely , we analyze with the PAC-Bayesian framework the expected risk of a MV drawn from the posterior distribution of MVs . The main difference with the literature is that we propose a stochastic MV , while previous works aim at studying randomized evaluations of the true risk of the deterministic MV . Doing so , we are able to derive tight empirical PAC-Bayesian bounds for our model directly on its expected risk , in Section 4 . We further propose , in Section 3 two approaches for optimizing the generalization bounds , hence learning the optimal posterior : the first optimizes an analytical and differentiable form of the empirical risk that can be derived when considering Dirichlet distributions ; the second optimizes a Monte Carlo approximation of the expected risk and can be employed with any form of posterior . In our experiments , reported in Section 5 , we first compare these two approaches , highlighting in which regimes one is preferable to the other . Finally , we assess our method ’ s performance on real benchmarks w.r.t . the performance of PAC-Bayesian approaches also learning MV classifiers . These results indicate that our models enjoy generalization bounds that are consistently tight and non-vacuous both when studying ensembles of data-independent predictors and when studying ensembles of data-dependent ones . Societal impact . Our work abides by the ethical guidelines enforced in contemporary research in machine learning . Given the theoretical nature of our contributions we do not foresee immediate potentially negative societal impact . 2 Notation and background . In this section , we formally define weighted Majority Vote ( MV ) classifiers and review the principal PAC-Bayesian approaches for learning them . 2.1 Weighted majority vote classifiers . Consider the data random variable pX , Y q , taking values in XˆY with X Ď Rd a d-dimensional representation space and Y the set of labels . We denote P the ( unknown ) data distribution of pX , Y q . We define a set ( dictionary ) of base classifiers D “ thj : X Ñ YuMj “ 1 . The weighted majority vote classifier is a convex combination of the base classifiers from D. Formally , a MV is parameterized by a weight vector θ P r0 , 1sM , such that řM j “ 1 θj “ 1 hence lying in the ( M -1 ) -simplex ∆M´1 , as follows : fθpxq “ argmax yPY M ÿ j “ 1 θj 1phjpxq “ yq , ( 1 ) where 1p¨q is the indicator function . Let WθpX , Y q be the random variable corresponding to the total weight assigned to base classifiers that predict an incorrect label on pX , Y q , that is WθpX , Y q “ M ÿ j “ 1 θj1phjpXq ‰ Y q . ( 2 ) In binary classification with |Y| “ 2 , the MV errs whenever WθpX , Y q ě 0.5 [ Lacasse et al. , 2010 , Masegosa et al. , 2020 ] . Hence the true risk ( w.r.t . 01-loss ) of the MV classifier can be expressed as Rpfθq fi EP 1pWθpX , Y q ě 0.5q “ PpWθ ě 0.5q . ( 3 ) Similarly , the empirical risk of fθ on a n-sample S “ tpxi , yiq „ Puni “ 1 is given by R̂pfθq “ n ÿ i “ 1 1pWθpxi , yiq ě 0.5q . Note that the results we introduce in the following are stated for binary classification , but are valid also in the multi-class setting ( |Y|ą2 ) . Indeed , in this context Equation ( 3 ) becomes a surrogate of the risk : we have Rpfθq ď PpWθ ě 0.5q ( see the notion of ω-margin with ω “ 0.5 proposed by Laviolette et al . [ 2017 ] ) . 2.2 A PAC-Bayesian perspective on the majority vote . The current paper stands in the context of PAC-Bayesian MV learning . The PAC-Bayesian framework has been employed to study the generalization guarantees of randomized classifiers . It is known to provide tight risk certificates that can be used to derive self-bounding learning algorithms [ e.g . Germain et al. , 2009 , Dziugaite and Roy , 2017 , Pérez-Ortiz et al. , 2020 ] . In the PAC-Bayes literature of the analysis of MV , θ is interpreted as the parameter of a categorical distribution Cpθq over the set of base classifiers D [ e.g . Germain et al. , 2015 , Lorenzen et al. , 2019 , Masegosa et al. , 2020 ] . In this sense , fθ corresponds to the MV predictor fθpxq “ argmaxyPY Eh „ Cpθq1phpxq “ yq and Wθ corresponds to the expected ratio of wrong predictors WθpX , Y q “ Eh „ Cpθq1phpXq ‰ Y q . The PAC-Bayesian analysis provides a sensible way to find such a categorical distribution , called posterior distribution , that leads to a model with low true risk Rpfθq . However , an important caveat is that the PAC-Bayesian generalization bounds can not be derived directly on the true risk Rpfθq , without making assumptions on the distribution of Wθ and raising fidelity problems . Thus , a common approach is to consider oracle1 upper bounds on the true risk in terms of quantities from which PAC-Bayesian bounds can be derived , that are typically related to the statistical moments of Wθ . By doing so , oracle bounds act as a proxy for estimating the cumulative density function for Wθ ě 0.5 [ Langford and Shawe-Taylor , 2002 , Germain et al. , 2006 , Lacasse et al. , 2010 , Masegosa et al. , 2020 ] . Generalization guarantees for Rpfθq are hence derived , involving the empirical counterpart of the oracle bound , the KL-divergence between the posterior categorical distribution Cpθq and a prior one . An overview of the existing oracle bounds described below is represented in Figure 1 . First order bound . The most classical “ factor two ” oracle bound [ Langford and Shawe-Taylor , 2002 ] is derived considering the relation between MV ’ s risk and the first moment of Wθ , a.k.a . the expected risk of the randomized classifier : R1pθq fi Eh „ Cpθq1phpXq‰Y q “ EP Wθ . 1Oracle bounds are expressed in terms of the unknown data distribution ; their exact value can not be computed . By applying Markov ’ s inequality , we have Rpfθq ď 2R1pθq . This “ factor two ” bound is close to zero only if the expectation of the risk of a single base classifier drawn according to Cpθq is itself close to zero . Therefore , it does not take into account the correlation between base predictors , which is key to characterize how a MV classifier can achieve strong predictions even when its base classifiers are individually weak ( as observed when performing e.g . Boosting [ Schapire and Singer , 1999 ] ) . This explains why R1pθq can be a very loose estimate of Rpfθq when the base classifiers are adequately decorrelated . Binomial bound . A generalization of the first order approach was proposed in Shawe-Taylor and Hardoon [ 2009 ] , Lacasse et al . [ 2010 ] , where the true risk of the MV is estimated by drawing multiple ( N ) base hypotheses and computing the probability that at least N2 make an error ( which is given by the binomial with parameter Wθ ) : WθN pX , Y q fi N ÿ k “ N2 ˆ N k ˙ W kθ p1´WθqpN´kq . The higher N , the better WθN pX , Y q approximates the true risk , but the looser the bound , as the KL term worsens by a factor of N . Moreover , with this approach it is not possible to derive generalization bounds directly on the true risk , and the corresponding oracle bound still presents a factor two : Rpfθq ď 2 EPWθN . Second order bound . A parallel line of works focuses on improving the bounds by accounting for voter correlations , i.e . considering the agreement and/or disagreement of two random voters . Masegosa et al . [ 2020 ] recently proposed a new upper bound for the true risk depending on the second moment of Wθ , a.k.a . tandem loss or joint error , the risk that two predictors make a mistake on the same point : R2pθqfiEP Eh „ Cpθq , h1 „ Cpθq1phpXq‰Y ^h1pXq‰Y q “ EP W 2θ . By applying the second order Markov ’ s inequality , we have Rpfθqď4R2pθq . Masegosa et al . [ 2020 ] show that in the worst case ( i.e . when the base classifiers are identical ) the second order bound could be twice worse than the first order bound ( R2pθq « 2R1pθq ) , while in the best case ( i.e . the M base classifiers are perfectly decorrelated ) , the second order bound could be an order of magnitude tighter ( R2pθq « 1MR1pθq ) . C-bound . Originally proposed by Breiman [ 2001 ] in the context of Random Forest , a bound known as the C-Bound in the PAC-Bayesian literature [ Lacasse et al. , 2006 , Roy et al. , 2011 , Germain et al. , 2015 , Laviolette et al. , 2017 , Viallard et al. , 2021b ] is derived by considering explicitly the joint error and disagreement between two base predictors . Using Chebyshev-Cantelli ’ s inequality , the C-Bound can be written in terms of the first and second moments of Wθ : Rpfθq ď R2pθq ´ Eh „ Cpθq pEP1phpXq ‰ Y qq 2 R2pθq ´R1pθq ` 14 . This bound is tighter than the second order one if the disagreement Eh , h1EPp1phpxq ‰ h1pXqqq is greater than R2pθq [ Masegosa et al. , 2020 ] . From a practical point of view , one could minimize the generalization bounds of one of the above methods to learn a weight distribution over an ensemble of predictors . However , this could lead to suboptimal MV classifiers . To illustrate this behavior we plot in Figure 3 the decision surfaces learnt by the minimization of a PAC-Bayesian bound on each of the aforementioned oracle bounds . These plots provide evidence that , when the base classifiers are weak , state-of-the-art PAC-Bayesian methods do not necessarily build powerful ensembles ( failing to improve upon a Naive Bayes approach [ Berend and Kontorovich , 2015 ] ) . First Order concentrates on few base classifiers , as previously observed by Lorenzen et al . [ 2019 ] and Masegosa et al . [ 2020 ] , while Second Order and C-Bound fail to leverage the diversity in the whole set of classifiers . Indeed , in this setting the base classifiers are weak , but diverse enough so that there exists an optimal combination of them that perfectly splits the two classes without error . However , the optimization of the PAC-Bayes guarantees over Second Order and C-Bound are shown to select a small subset of base classifiers which is not enough to achieve good performance . On the contrary , Binomial is able to fit the problem by drawing more than 2 voters , but it provides generalization bounds that are loose even when the learned model exhibits good generalization capabilities , as in this case .

Summary: The authors propose an algorithm for learning a *stochastic* weighted majority vote classifier, by optimizing a (PAC-Bayesian) generalization bound. They consider Dirichlet distributions over classifiers as well as general measures, and derive analytical and numerical schemes, respectively, for optimizing both. They empirically evaluate the algorithm for several datasets and model classes (decision stumps and random forest). They also compare against deterministic weighted majority vote classifiers learned by optimizing Gibbs classifiers via existing PAC-Bayes bounds (first order, secord order, C-bounds, etc). The resulting stochastic predictor achieves competitive performance and bound tightness compared to ones obtained by minimizing alternative bounds.