On the Geometry of Deep Bayesian Active Learning

1 INTRODUCTION . Lack of training labels restricts the performance of deep neural networks ( DNNs ) , though prices of GPU resources were falling fast . Recently , leveraging the abundance of unlabeled data has become a potential solution to relieve this bottleneck whereby expert knowledge is involved to annotate those unlabeled data . In such setting , the deep learning community introduced active learning ( AL ) ( Gal et al. , 2017 ) that , maximizing the model uncertainty ( Ashukha et al. , 2019 ; Lakshminarayanan et al. , 2017 ) to acquire a set of highly informative or representative unlabeled data , and solicit experts ’ annotations . During this AL process , the learning model tries to achieve a desired accuracy using minimal data labeling . Recent shift of model uncertainty in many fields , such as Bayesian neural networks ( Blundell et al. , 2015 ) , Monte-Carlo ( MC ) dropout ( Gal & Ghahramani , 2016 ) , and Bayesian core-set construction ( Sener & Savarese , 2018 ) , shows that , new scenarios arise from deep Bayesian AL ( Pinsler et al. , 2019 ; Kirsch et al. , 2019 ) . Bayesian AL ( Golovin et al. , 2010 ; Jedoui et al. , 2019 ) presents an expressive probabilistic interpretation on model uncertainty ( Gal & Ghahramani , 2016 ) . Theoretically , for a simple regression model such as linear , logistic , and probit , AL can derive their closed-forms on updating one sparse subset that maximally reduces the uncertainty of the posteriors over the regression parameters ( Pinsler et al. , 2019 ) . However , for a DNN model , optimizing massive training parameters is not easily tractable . It is thus that Bayesian approximation provides alternatives including importance sampling ( Doucet et al. , 2000 ) and Frank-Wolfe optimization ( Vavasis , 1992 ) . With importance sampling , a typical approach is to express the information gain in terms of the predictive entropy over the model , and it is called Bayesian active learning by disagreements ( BALD ) ( Houlsby et al. , 2011 ) . BALD has two interpretations : model uncertainty estimation and core-set construction . To estimate the model uncertainty , a greedy strategy is applied to select those data that maximize the parameter disagreements between the current training model and its subsequent updates as ( Gal et al. , 2017 ) . However , naively interacting with BALD using uninformative prior ( Strachan & Van Dijk , 2003 ) ( Price & Manson , 2002 ) , which can be created to reflect a balance among outcomes when no information is available , leads to unstable biased acquisitions ( Gao et al. , 2020 ) , e.g . insufficient prior labels . Moreover , the similarity or consistency of those acquisitions to the previous acquired samples , brings redundant information to the model and decelerates its training . Core-set construction ( Campbell & Broderick , 2018 ) avoids the greedy interaction to the model by capturing characteristics of the data distributions . By modeling the complete data posterior over the distributions of parameters , BALD can be deemed as a core-set construction process on a sphere ( Kirsch et al. , 2019 ) , which seamlessly solicits a compact subset to approximate the input data distribution , and efficiently mitigates the sensitivity to uninformative prior and redundant information . From the view of geometry , updates of core-set construction is usually optimized with sphere geodesic as ( Nie et al. , 2013 ; Wang et al. , 2019 ) . Once the core-set is obtained , deep AL immediately seeks annotations from experts and starts the training . However , data points located at the boundary regions of the distribution , usually win uniform distribution , can not be highly-representative candidates for the core-set . Therefore , constructing the coreset on a sphere may not be the optimal choice for deep AL . This paper presents a novel AL framework , namely Geometric BALD ( GBALD ) , over the geometric interpretation of BALD that , interpreting BALD with core-set construction on an ellipsoid , initializes an effective representation to drive a DNN model . The goal is to seek for significant accuracy improvements against an uninformative prior and redundant information . Figure 1 describes this two-stage framework . In the first stage , geometric core-set construction on an ellipsoid initializes effective acquisitions to start a DNN model regardless of the uninformative prior . Taking the core-set as the input features , the next stage ranks the batch acquisitions of model uncertainty according to their geometric representativeness , and then solicits some highly-representative examples from the batch . With the representation constraints , the ranked acquisitions reduce the probability of sampling nearby samples of the previous acquisitions , preventing redundant acquisitions . To guarantee the improvement , our generalization analysis shows that , the lower bound of generalization errors of AL with the ellipsoid is proven to be tighter than that of AL with the sphere . Achieving a nearly zero generalization error by AL with ellipsoid is also proven to have higher probability . Contributions of this paper can be summarized from Geometric , Algorithmic , and Theoretical perspectives . • Geometrically , our key innovation is to construct the core-set on an ellipsoid , not typical sphere , preventing its updates towards the boundary regions of the distributions . • In term of algorithm design , in our work , from a Bayesian perspective , we propose a two-stage framework that sequentially introduces the core-set representation and model uncertainty , strengthening their performance “ independently ” . Moreover , different to the typical BALD optimizations , we present geometric solvers to construct core-set and estimate model uncertainty , which result in a different view for Bayesian active learning . • Theoretically , to guarantee those improvements , our generalization analysis proves that , compared to typical Bayesian spherical interpretation , geodesic search with ellipsoid can derive a tighter lower error bound and achieve higher probability to obtain a nearly zero error . See Appendix B . The rest of this paper is organized as follows . In Section 2 , we first review the related work . Secondly , we elaborate BALD and GBALD in Sections 3 and 4 , respectively . Experimental results are presented in Section 5 . Finally , we conclude this paper in Section 6 . 2 RELATED WORK . Model uncertainty . In deep learning community , AL ( Cohn et al. , 1994 ) was introduced to improve the training of a DNN model by annotating unlabeled data , where the data which maximize the model uncertainty ( Lakshminarayanan et al. , 2017 ) are the primary acquisitions . For example , in ensemble deep learning ( Ashukha et al. , 2019 ) , out-of-domain uncertainty estimation selects those data which do not follow the same distribution as the input training data ; in-domain uncertainty draws the data from the original input distribution , producing reliable probability estimates . Gal & Ghahramani ( 2016 ) use MC dropout to estimate predictive uncertainty for approximating a Bayesian convolutional neural network . Lakshminarayanan et al . ( 2017 ) estimate predictive uncertainty using a proper scoring rule as the training criteria to fed a DNN . Bayesian AL . Taking a Bayesian perspective ( Golovin et al. , 2010 ) , AL can be deemed as minimizing the Bayesian posterior risk with multiple label acquisitions over the input unlabeled data . A potential informative approach is to reduce the uncertainty about the parameters using Shannon ’ s entropy ( Tang et al. , 2002 ) . This can be interpreted as seeking the acquisitions for which the Bayesian parameters under the posterior disagree about the outcome the most , so this acquisition algorithm is referred to as Bayesian active learning by disagreement ( BALD ) ( Houlsby et al. , 2011 ) . Deep AL . Recently , deep Bayesian AL attracted our eyes . Gal et al . ( 2017 ) proposed to cooperate BALD with a DNN to improve the training . The unlabeled data which maximize the model uncertainty provide positive feedback . However , it needs to repeatedly update the model until the acquisition budget is exhausted . To improve the acquisition efficiency , batch sampling with BALD is applied as ( Kirsch et al. , 2019 ; Pinsler et al. , 2019 ) . In BatchBALD , Kirsch et al . ( 2019 ) developed a tractable approximation to the mutual information of one batch of unlabeled data and current model parameters . However , those uncertainty evaluations of Bayesian AL whether in single or batch acquisitions all take greedy strategies , which lead to computationally infeasible , or excursive parameter estimations . For deep Bayesian AL , being short of interactions to DNN can not maximally drive their model performance as ( Pinsler et al. , 2019 ; Sener & Savarese , 2018 ) , etc . 3 BALD . BALD has two different interpretations : model uncertainty estimation and core-set construction . We simply introduce them in this section . 3.1 MODEL UNCERTAINTY ESTIMATION . We consider a discriminative model p ( y∣x , θ ) parameterized by θ that maps x ∈ X into an output distribution over a set of y ∈ Y . Given an initial labeled ( training ) set D0 ∈ X × Y , the Bayesian inference over this parameterized model is to estimate the posterior p ( θ∣D0 ) , i.e . estimate θ by repeatedly updating D0 . AL adopts this setting from a Bayesian view . With AL , the learner can choose unlabeled data from Du = { xi } Nj=1 ∈ X , to observe the outputs of the current model , maximizing the uncertainty of the model parameters . Houlsby et al . ( 2011 ) proposed a greedy strategy termed BALD to update D0 by estimating a desired data x∗ that maximizes the decrease in expected posterior entropy : x∗ = arg max x∈Du H [ θ∣D0 ] −Ey∼p ( y∣x , D0 ) [ H [ θ∣x , y , D0 ] ] , ( 1 ) where the labeled and unlabeled sets are updated by D0 = D0 ∪ { x∗ , y∗ } , Du = Du/x∗ , and y∗ denotes the output of x∗ . In deep AL , y∗ can be annotated as a label from experts and θ yields a DNN model . 3.2 CORE-SET CONSTRUCTION . Let p ( θ∣D0 ) be updated by its log posterior logp ( θ∣D0 , x∗ ) , y∗ ∈ { yi } Ni=1 , assume the outputs are conditional independent of the inputs , i.e . p ( y∗∣x∗ , D0 ) = ∫θ p ( y ∗∣x∗ , θ ) p ( θ∣D0 ) dθ , then we have the complete data log posterior following ( Pinsler et al. , 2019 ) : Ey∗ [ logp ( θ∣D0 , x∗ , y∗ ) ] = Ey∗ [ logp ( θ∣D0 ) + logp ( y∗∣x∗ , θ ) − logp ( y∗∣x∗ , D0 ) ] = logp ( θ∣D0 ) +Ey∗ [ logp ( y∗∣x∗ , θ ) +H [ y∗∣x∗ , D0 ] ] = logp ( θ∣D0 ) + N ∑ i=1 ⎛ ⎝ Eyi [ logp ( yi∣xi , θ ) +H [ yi∣xi , D0 ] ] ⎞ ⎠ . ( 2 ) The key idea of core-set construction is to approximate the log posterior of Eq . ( 2 ) by a subset of D′u ⊆ Du such that : EYu [ logp ( θ∣D0 , Du , Yu ) ] ≈ EY ′u [ logp ( θ∣D0 , D ′ u , Y ′ u ) ] , where Yu and Y ′ u denote the predictive labels of Du and D′u respectively by the Bayesian discriminative model , that is , p ( Yu∣Du , D0 ) = ∫θ p ( Yu∣Du , θ ) p ( θ∣D0 ) dθ , and p ( Y ′ u∣D ′ u , D0 ) = ∫θ p ( Y ′ u∣D ′ u , θ ) p ( θ∣D0 ) dθ . Here D′u can be indicated by a core-set ( Pinsler et al. , 2019 ) that highly represents Du . Optimization tricks such as Frank-Wolfe optimization ( Vavasis , 1992 ) then can be adopted to solve this problem . Motivations . Eqs . ( 1 ) and ( 2 ) provide the Bayesian rules of BALD over model uncertainty and core-set construction respectively , which further attract the attention of the deep learning community . However , the two interpretations of BALD are limited by : 1 ) redundant information and 2 ) uninformative prior , where one major reason which causes these two issues is the poor initialization on the prior , i.e . p ( D0∣θ ) . For example , unbalanced label initialization on D0 usually leads to an uninformative prior , which further conducts the acquisitions of AL to select those unlabeled data from one or some fixed classes ; highly-biased results with ( Gao et al. , 2020 ) redundant information are inevitable . Therefore , these two limitations affect each other . 4 GBALD GBALD consists of two components : initial acquisitions based on core-set construction and model uncertainty estimation with those initial acquisitions .

Summary: This paper is basically unreadable. The sentence structure / grammar is strange, and if that was the only issue it could be overlooked. The paper also does not describe or explain the motivation and interpretation of anything, but instead just lists equations. For example, eta is the parameter that projects a spherical geodesic onto an the ellipsoid one, and an ellipsoid geodesic prevents updates of the core-set towards the boundary regions where the characteristics of the distribution cannot be captured. However, what are these characteristics, and how can they motivate how to choose eta?